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CHAPTER  1 


INTRODUCTION 

In  this  paper  two  different  types  of  discrete  time  Markov 
maintenance  models  having  repair  facilities  available  will  be  investigated. 
Because  of  their  wide  applicability  in  the  practical  world,  a number  of 
authors  have  studied  optimization  problems  for  discrete  time  machine 
maintenance  models.  Their  main  concern  has  been  on  the  structure  of  an 
optimal  policy,  and  a simple  repair  rule  called  a control  limit  policy 
has  been  introduced.  When  there  is  an  operating  machine,  with  its  con- 
dition deteriorating  as  time  goes  on,  a control  limit  policy  is  defined 
as  a policy  where  a repair  or  replacement  action  is  taken  if  and  only 
if  the  degree  of  deterioration  of  the  operating  machine  exceeds  some 
critical  value  Control  limit  policies  are  intuitively  appealing 
policies,  and  sufficient  conditions  under  which  a control  limit  policy 
is  optimal  have  been  derived 

In  1963  Derman  [2]  introduced  the  basic  maintenance  model  of  this 
type.  In  his  model,  two  actions  are  available  at  each  time  period,  i.e. 
replace  the  current  machine,  or  leave  the  current  machine  as  is.  If  a 
replacement  action  is  taken,  a new  machine  replaces  the  old  one,  and 
the  new  machine  begins  to  operate  in  its  best  condition  at  the  beginning 
of  the  next  period.  If  the  nonreplacement  action  is  chosen,  the  machine 
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keeps  operating,  and  its  degree  of  deterioration  evolves  from  i to  j 
in  one  period  with  transition  probability  p^,  0 < i,  j < I.  He  developed 
the  notion  of  an  increasing  failure  rate  (IFR)  distribution,  which  plays 
an  important  role  in  reliability  theory  (see  Barlow  [1]),  for  a Markov 
chain  with  a finite  number  of  states.  A Markov  chain  is  said  to  be  IFF 
if  a higher  state  is  associated  with  a greater  chance  of  further 
deterioration.  More  precisely,  a Markov  chain  with  the  transition  matrix 
(p^j)  is  said  to  be  IFR  if  P^-)  is  stochastically  smaller  than  P^+^(*) 
for  0 < i < 1-1,  and  can  be  written  as 

Pi(>)  c Pi+1(,)  ’ 0 < 1 < 1-1, 

where 

P.  (k)  = E p 0 < i < I . 

3<k 

Note  that  if  F(t)  and  G(t)  are  distribution  functions,  F(»)  e G(-) 
if  and  only  if  F(t)  > G(t)  for  any  t.  Assuming  a simple  cost  structure 
and  the  IFR  property  on  the  transition  probabilities,  Derman  showed  the 
optimality  of  a control  limit  policy. 

Kolesar  [g]  extended  the  basic  model  by  introducing  state  depen- 
dent operating  costs,  and  Kalymon  [6]  further  generalized  the  cost 
structure  by  allowing  replacement  costs  to  be  stochastic.  Klein  [8] 
expanded  Derman' s model  to  include  a costly  inspection.  Eppen  [4], 

Taylor  [l4],  and  Rosenfield  [12]  developed  their  models  along  this  line 
with  emphasis  on  trying  to  find  simple  types  of  policies  which  optimize 
their  models. 
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A joint  replacement  and  stocking  problem  was  introduced  by  Derman 
and  Lieberman  [3j,  which  was  generalized  by  Ross  [13]  to  allow  for 
deterioration  of  a machine. 

The  aforementioned  models  have  the  properties  that  the  amount 
of  time  needed  for  the  repair  of  a machine  is  one  unit  of  time,  and  an 
unlimited  supply  of  new  spares  is  available.  In  that  sense,  they  are 
replacement  models  rather  than  repair  models.  In  1973  Kao  [7]  intro- 
duced a semi -Markovian  approach  to  the  basic  model.  In  his  model,  the 
repair  time  of  a machine  is  no  longer  instantaneous  but  takes  some  random 
time  according  to  its  semi-Markovian  nature,  while  the  supply  of  new 
spares  is  kept  unlimited.  Hatoyama  [5]  investigated  some  machine  repair 
models  assuming  that  the  repair  time  of  a machine  has  a geometric  dis- 
tribution. A model  with  finite  number  of  spare  machines  was  also  con- 
sidered there. 

In  Chapter  2 of  this  paper,  the  basic  machine  repair  problem 
is  extended  to  allow  for  the  existence  of  several  repair  shops.  The 
repair  time  distributions  depend  on  the  type  of  repair  work  required 
on  a machine  rather  than  its  condition..  Here  the  condition  of  the  machine 
regulates  the  type  of  repair  work  required  in  a probabilistic  manner 
rather  than  deterministically.  This  extension  seems  suitable  for  the 
maintenance  problems  where  the  state  of  the  machine  such  as  its  age 
does  not  completely  coincide  with  the  type  of  repair  work  on  it.  In  a 
manner  similar  to  that  presented  for  the  type  of  models  treated  in  [5] 
control  limit  policy  is  defined,  and  sufficient  conditions  for  the 
optimality  of  such  control  limit  policies  are  obtained. 


Chapter  3 considers  the  machine  repair  problem  in  tne  context 
of  optimal  control  of  queues.  Since  the  number  of  machines  in  the 
system  is  fixed  and  finite,  the  models  treated  here  necessarily  become 
discrete  time  closed  queueing  systems.  There  appears  to  have  been 
almost  no  research  in  optimization  of  discrete  time  closed  queueing 
systems.  Discrete  time  queueing  systems  have  been  investigated  by 
Magazine  1 10 J , [11],  but  his  models  are  open  queueing  systems.  Models 
for  the  optimal  control  of  closed  queueing  systems  have  been  studied 
by  Torbett  [15],  but  his  analysis  is  time  continuous.  In  our  models, 
the  decision  maker  has  an  option  of  opening  or  closing  the  repair  shop 
when  there  are  machines  waiting  for  repair  service,  as  well  as  the 
option  of  repairing  or  leaving  an  operating  machine  alone.  A set  up 
cost  for  opening  a closed  gate,  a shut  down  cost  for  closing  an  open 
gate,  a holding  cost  for  holding  machines  in  the  repair  system,  and  a 
service  cost  for  operating  an  open  repair  shop  are  introduced  as  is 
done  in  the  usual  queueing  control  models.  A two-dimensional  control 
limit  policy  is  defined,  2nd  sufficient  conditions  for  the  existence 

J 

of  such  a policy  minimizing  the  total  expected  a-discounted  cost  are 
derived. 

I 

i 

i 
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CHAPTER  2 


MULTI -REFAIR -TYPE  MAINTENANCE  MODELS 

This  chapter  treats  maintenance  models  when  there  is  more  than 
one  possible  type  of  repair  in  the  system.  Here,  the  repair  time  distri- 
butions, material  cost,  and  labor  cost  all  depend  on  the  type  of  repair 

work  required  on  the  machine.  Sufficient  conditions  which  result  in 

% 

the  optimality  of  control  limit  policies  of  some  kind  are  obtained.  A 
generalized  model  is  then  considered.  Finally,  a model  where  there 
is  one  repairman  for  each  type  of  repair  shop  is  studied. 

2. 1.  The  Model 

We  consider  a basic  multi-repair-type  Markov  machine  maintenance 
model.  The  flow  of  machines  in  the  system  is  schematically  shown  in  Fig.  2.1. 

There  is  an  operating  machine  and  S 
(S  > 1)  spare  machines  in  the  system. 

The  system  is  observed  periodically  and 
at  the  beginning  of  each  period  an 
operating  machine  is  classified  as 
being  in  one  of  1+1  (i  > 1)  states, 
with  each  state  showing  the  degree  of 
deterioration,  0 represents  a machine 
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Figure  2.1.  A Multi-Repair-Type 
Machine  Maintenance 
System. 


in  its  best  condition,  while  1 denotes  its  failure.  When  a machine 
is  operating,  two  choices  are  available  at  the  beginning  of  each  period: 
to  let  it  keep  operating,  or  to  repair  it.  If  the  former  decision  is 
chosen,  the  state  of  the  operating  machine  evolves  from  i to  j in 
one  unit  of  time  according  to  the  transition  probability  p_  > 0. 

If  the  latter  is  selected,  the  machine  is  immediately  sent  to  one  of 
K (K  > 2)  repair  shops,  and  is  replaced  by  one  of  the  spare  machines, 
if  any  is  available.  The  new  operating  machine  begins  to  operate  in 
its  best  condition.  Type  1 through  type  K repair  shops  are  arranged 
so  that  type  1 represent"  ie  easiest  repair  work  and  type  K the 
hardest.  If  a machine  in  the  i-th  operating  condition  is  chosen  to 
be  repaired,  a type  k repair  facility  is  required  with  probability 

machine  must  be  repaired.  Furthermore,  suppose  the  type  of  repair 
required  if  a repair  decision  is  made  is  known  to  the  decision  maker 
before  he  takes  an  action.  Then  the  total  information  available  to 
the  decision  maker  at  the  beginning  of  each  period  is  the  condition  of 
an  operating  machine,  the  type  of  repair  work  required  if  the  repair 
decision  is  chosen,  and  the  number  of  machines  in  the  type  k repair 
shop  for  each  1 < k < K.  Later,  the  case  where  the  type  of  repair 
work  required  is  not  known  to  the  decision  maker  will  be  considered. 

If  a machine  in  the  i-th  operating  condition  is  sent  to  the 
type  k (l  < k < K)  repair  shop,  the  repair  work  starts  immediately, 
and  its  repair  time  T^,  which  is  independent  of  i,  is  assumed  to  be 
a randon)  variable  having  a geometric  distribution  with  parameter  q^, 
i.e., 


K ( i ) 

> 0.  Here,  we  assume  p^ 


0 < i < I.  A failed 
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j =1,2,...  . 


p{Tk = Jl  = 

If  all  the  machines  are  in  the  repair  shops,  the  system  fails 
since  no  machine  is  available.  In  that  case  we  must  wait  until  one 
of  the  machines  is  completely  repaired.  A penalty  cost,  P,  is  assessed 
per  period  during  the  system’s  failure,  A(i)  is  the  operating  cost 
for  a machine  in  the  i-th  operating  condition,  C(i,k)  is  the  material 
cost  for  repairing  a machine  in  the  i-th  operating  condition  in  the  type  k 
repair  shop,  and  B(k)  is  the  labor  cost  for  the  type  k repair  work 
for  a single  machine  per  period.  The  objective  is  to  find  a repair 
policy  which  minimizes  the  total  a-discounted  cost. 

Before  proceeding  further  it  is  useful  to  give  an  example  in 
which  the  above  model  can  be  applied.  Consider  the  problem  of  a taxicab 
driver  who  owns  several  cars.  He  continues  to  use  one  of  his  cars  until 
he  decides  to  repair  it.  He  then  begins  to  use  another  one,  if  any  is 
available.  We  assume  that  the  maintenance,  or  operating  cost  (excluding 
repair  costs),  of  a car  depends  mainly  on  its  age  or  the  number  of  days 
after  its  last  repair.  However,  the  actual  deterioration  of  the  car 
does  not  necessarily  coincide  with  its  age  Among  cars  having  the  same 
age,  some  may  require  little  repair  work,  while  others  may  require 
substantial  amounts  of  repair  work,,  A car  with  i-th  age  requires 
type  k repair  work  with  probability  p^.  If  the  taxicab  driver  is 
familiar  with  the  repair  Job,  he  can  get  the  information  on  the  type 
of  repair  work  required  as  well  as  the  age  of  the  car  before  he  makes 
the  decision  on  whether  or  not  the  car  should  be  repaired.  Otherwise 
he  can  utilize  just  the  age  of  the  car  when  he  makes  a decision. 
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The  repair  time  distribution  is  assumed  to  be  geometric,  and  its 
parameter  depends  on  the  type  of  repair  work.  Also,  repair  costs 
mainly  depend  on  the  type  of  repair  work.  If  all  the  cars  are  in  the 
repair  facilities,  no  car  is  available  to  him,  and  hence,  he  loses  some 
amount  of  the  expected  revenue.  If  he  is  interested  in  the  total  /-dis- 
counted cost,  then  finding  the  best  repair  schedule  is  an  example  of 
the  type  of  problem  which  will  be  treated  in  this  chapter. 

The  state  of  this  system  is  represented  by  the  K+2  vector 


(i,k, a^f . . . , s ) 


where  at  the  beginning  of  the  t-th  period  there  is  an  operating  machine 
whose  operating  condition  is  i,  type  k repair  work  is  required  if  the 
repair  decision  is  chosen,  and  s.  (l  < j < K)  machines  are  in  the 
type  j repair  facility.  Here 

K 

0 < i < I,  1 < k < K,  0 < E s . < S,  s > 0 (1  < J < K)  . 

j=l  J J ~ 

For  notational  convenience,  we  define 

K 

X+  = (0,1,8..,...,  s),  Z s = S+l,  s > 0 (1  < j < K) 

X in  J J - — - 

if  all  the  machines  are  in  the  repair  facilities  and  none  of  the 
machines  is  operating  at  the  beginning  of  the  t-th  period. 

In  order  to  simplify  the  repetitive  use  of  the  following  sets, 

define 
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es5  = fi|0  < i < I»  i ••  integer) 
/\  = |'k|l  < k < K,  k integer) 


XV 

Am  = f(s1,...,sv)l  L s.  < m,  s.  (l  < j < K)  ; nonnegative  integer) 
° 1 K-  ^=1  K * J 

K 

Jq  = {(0,1,sl,  ...,sr)|  L sk=m,  Sj  (1<0<K)  : nonnegative  integer). 

k*-l 


U) 

ss 


Let  qw/  be  the  probability  that  s (s  < s)  machines  are 


still  in  the  type  o repair  shop  at  the  end  of  the  period,  given  s 
machines  are  in  the  type  j (l  < j < K)  repair  shop  at  the  beginning  of 
a period.  Then 


1 < j < K , 


since  the  repair  time, 


has  a geometric  distribution  with  parameter 


Before  formulating  the  problem  as  a dynamic  programming 
problem,  we  make  the  following  simplification.  When  a new  machine 
replaces  the  previously  operating  machine,  the  new  one  starts  operating 
in  its  best  condition  at  tne  beginning  of  the  next  period,  and  the 
least  repair  work,  i e.,  type  1 repair  is  required  if  it  is  instantaneously 
determined  to  be  repaired. 

Let  V (i,k,s.., . ..,s„;n)  be  the  minimum  expected  n period 

a-discounted  cost  starting  from  state  (i, k, s^, . . . , s^) . Then  by  setting 

Va(i,k,s1,...,sK;0)  = 0 for  any  feasible  (i,k, s^ . . . , sR) , 

V (i,k,s.,...,s1.;n)  (n  > 1)  satisfies  the  following  set  of  recursive 
OC  1.  K. 

equations: 
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i » It*  s ^ , . . t>s^;n) 


K I K 1 K 

--  minfA(i)  + ZB(j)s.  +a  £ £ £ •••  £ p , 

j = l J i ' -0  k'  =1  s.K)  s' =0  11 

1 K 

• pCi,)q(l)  ...a(K)  V (i-  k'  s'  s'-n  1) 

Fk’  qs.s;  sv s'  vat'1  ,K  >si>--->sK>n  1)> 

11  A A 

K sp  3jcJ’1  SK 

C(i, k)  + B(k  +ZB(j)s.+a  £•••  £•■*  £ q ' 

-3*1  J s’=0  s^=0  s^=0  slsl 

•**  qVl,s'  qs^  Va(0’ L» si> • • • » SK> n-l) } 


for  (i,k,slf...,sK)  C J)  x 4£ 


(2.1) 


= P 


K S1  SK 

+ £ B(j)s  +a  £ •••  £ ![sll‘--q  Ks'  v (0,1,8*  ...,s';n-l) 

j=l  J S^-0  s^--=0  S151  SKSK  Q 1 K 


for  (l,k,sp...,sK)  € ^+1  . 
By  letting 


Ra(  s-^>  • 


f 


S1  SK 
£ •••  £ qU),. 

s}=0  s^O  slsl 


■q‘K\  V (i,k, s' , 
sKs^  a l’ 


• > n > , 


we  can  simplify  (2.1)  as  below:  For  n > 1, 
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! 


V ((i,k,  s^, . • • t Sj^j  n) 


K 

- min{A(i)  f Z B ( j ) s , +•  a 

j=l  J 


I K 

Z Z p 


i'-O  k' -1 


ii 


.P 


(*’) 

k’ 


R (i’ 

a 


k , s1. 


sK;n-l), 


K 

C(i,k)+B(k)+  Z B(j)s  +aR  : (0,1,8., 
j,l  J a 1 


sk-»-l,  ...,sK;  n-1)} 


for  (i.k.Sp  . . . ,sR)  6 ^ X j(x  (2.2) 

K 

= P+  Z B(j)s^  * Q3?a(0,l,s1, . . .,sK;n-l) 

3-1- 


for  (i,k,s][, 


) € ^ 


S+l 

0 


Let  V ( i,  k,  s^, . . . , Sj^)  be  the  total  expected  a-discounted  cost 
starting  from  state  (i,k,  . . . , sR)  6 x f(x  JS  U J^1  . Then 

k,  ai>  • • • > sr)  lim  V^(i,k,  s^,  „ . . ,s^n)  . 
n — » 00 

Furthermore,  the  existence  of  a stationary  policy  minimizing  the  total 
expected  /-discounted  cost  is  guaranteed  since  this  model  is  a Markov 
decision  process. 

Two  kinds  of  control  limit  policies  are  now  introduced. 


Definition.  An  i^  control  limit  policy  is  a nonrandomized  policy  where 
there  is  a special  operating  condition  i for  each  k (l  < k < K),  for 
each  feasib  e s - (s^, . . . , s^),  and  for  each  period  n (n  > 1),  say 
*k  s n’  such  that  for  a11  (i»k,s., ..  . ,s„)  with  i < i , the  decision 
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at  period  n is  to  keep  an  operating  machine  in  operation,  and  for  all 

(i, k, s, , . . . , s ) with  i > i the  decision  at  n is  to  repair  it. 

L K.  - K,  s,n 

A k control  limit  po]  icy  is  a nonrandomized  policy  wr.ere  there  is  a 

special  type  k of  repair  shop  for  each  i (0  < i v I),  for  each 

s (s, , . . . , s„) , and  for  each  period  n,  say  k.  , such  that  for  all 
IK  1 , s, n 

(i, k, s. , . . . , s ) with  k < k.  , the  decision  at  n is  to  repair 
i k ~ i , s,n 

an  operating  machine,  and  for  all  (i, k, s , . . . , s.  with  k > k. 

1 n 1 1 S j n 

the  decision  at  n is  to  keep  it  operating. 

Sufficient  conditions  for  the  existence  of  a stationary  i control 
limit  policy  which  minimizes  the  total  expected  G-discounted  cost  of  this 
model  are  obtained  first  through  several  lemmas  shown  next. 

Lemma  2. 1.  Assume  the  following  conditions  hold: 

1.  B(k)  is  nonnegative  and  nondecreasing  in  k for  1 < k < K. 

2.  qk  - qk+l  f°r  1 - k - K_1- 

3.  P > C(0,1). 

Then  for  (i,k'  ,s1, . . .,sK)  t J?  x f(x  1 U J n > 1, 

>a;  V (i,k  , s^, , . . , Sj^^+1, , . . , Sj^;  n ) y ^(i,k  , s^, . . . , s^+l, . . . , s^j  n ) . 

ib  VQ(i,k  > s^, . . . , Sj^+1, . . . , s^;  n)  ^ Va(i,k  , s^, . . . , s^, . . . , s^;  n • 

Proof.  Proof  is  by  mathematical  induction.  For  n = 1,  it  is  easy  to 
check  that  both  (a)  and  (b)  hold  for  (i , k' , s^, . . . , s ) x x 
and  the  proof  is  omitted.  Suppose  both  (a)  and  (b)  hold  for  n = m-1  > 1. 
Consider  the  case  for  n m.  Take  (a)  first. 
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r 


Notice  that  for  (i,k',Sl, . . . ,br)  € J)  x x J3'1  U J3, 

• * • , * * * / m-1 ) -R_^(i,k  , s^, . , . , s^+1, . . . , s^jm-l) 

-<i-Vl)  z-  z 

s^=0  s£=0  11  SK  K 

, s^, . . . , s^+^+l, . . . , Sj^; m-1 ) - V_^(i,  k' , s^, . . . , s^jm-l) ) 

- (1-q  ) Z ••*  Z q^,  •■•  q^, 

Sl=0  sk"°  11  K K 

> s^, . . . , s^+1, . . . , Sjj.;  m-1)  - V_^(i,  k' , s.^, . . . , s^j m-1) ) , 


by  Lemma  3.3  of  [5] 


S1  SK 


E-  s - £L 


s^=0  s£=0  “1“1 


K K 


■ ^ t s^»  • • • > sj^+i+^">  • • • » k' , s^, . . . , s^+1,  * " * ’ SK’ 

from  2 and  by  inductive  assumption  on  (b)  for  n = m-1 
> 0 , by  inductive  assumption  on  (a)  for  n = m-1. 

Using  the  above  result,  for  (l,k' ,s., . . .,s„)  , 

1 K U 

®1>  • • • > Sj^+i+l, . . . , Sr,  m)  - (0, 1,  s^, . . . , Sj^+1, . . . , s^j  m) 

K 

=P+  Z B(j)sj  + B(k+1)  + aRa(0,l,s1,...,sk+1+l, ...,sK;m-l) 

K 

- (P  + Z B(J)Sj  + B(k)  + aRa(0,l,Sl, ...,sk+l,...,sK,m-l)) 

13 


m-1) ) , 


= B(k+1)  - B(k)  + a(R^(0,l,sL,...,sk+1+l,  ...,sKjm-l) 

“ , 1 , s ^ , . . . , s k+l , . . . , s^, m -1  ) 


> 0,  from  1. 


For  (i,k'  ,8^  . . .,sK)  £ c$x  /fx  JS~  \ we  compare  the  correspond- 


ing values  term  by  term  ^see  [5]). 


Va'i,k  » si»  • • • »sk*i+1’ ' ’ ’ ,sK;m  1-st"  'Vc/i,k'  *si'  • * •»  sk+1>  • • • >sK>m'  )1.s. 
K IK 

= A(i)  + Z B(j)s  + B(k+1)  + a Z Z p,,,?'1 
j=l  J i’=0  j*l  J 

K 

*Ka(i’ , . . ,sk+1+l, . . .,sK;m-l)  - (A(i)  + Z B(j)s.  + B(k) 

j=l  J 

1 K U') 

+ Z • • • » • • • » ®— ^) ) 

I K 

= B(k+1)  - B(k) +a  Z Z p.-.p^1  ^(R  (i',j,s..,...,s  +l,...,s  ;m-l) 

i-=Oj=l  11  J a 1 X+1  K 

“ ^ , sk+l, . . , , s^j  m-1) ) 


> 0 . 


In  the  similar  manner,  we  have 


^ ^ ^ > s^>  • • • > si[+^+^»  • • •>  sK»m  ' ^2-nd"  vVa(i’k'  ’ ®i'  • • • » s^+^»  • • • » sj^J®)  ^2-r 


Hence  also  for  (i,  k' , s^  . . . , sR)  t Jt  x ft x J/5'1, 

• • • » ®k+^li  ■ • • »Sjjj®i  ^ ( i , k ' , s^, . . . , sk+l, . . . , Sj^j m) 


which  shows  that  (a)  holds  for  n = m. 


Consider  the  (b)  case  for  n * m.  For  (i,k' , s, , . . . , sj 

X K 

€«P  x /(x  \ we  again  compare  the  corresponding  values  term  by  term. 
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I 


' 


^Vc/i,k'  * sl' ' ■ ■ * sk+1'  • • • > sK;m)  >1_st  - <y  i,  k' , , sfc, . . . , sKJm)  >1_st 

I K 

■ 0(k)  + « ^ ^ Pii,Pji,)(Ra(i,,J,s1,...,sk+l,...,sK}m-l) 

” • • • * • • • > n-l) ) 

> 0,  from  1 and  inductive  assumption  on  (b)  for  n = m-1. 

Similarly,  the  difference  of  the  corresponding  second  terms  can 
be  shown  to  be  nonnegative. 

Lastly,  for  (i,  k’ , s^  . . . , sR)  £ d 

Va(0, 1»  s^, . • . , s^+1, . . . , s^j m)  - V^(0,  1,  sL, . . . , &k> . . . , Sj^;  m) 

K 

= P + Z B(  J ) s J + B(k)  + OfRa(0,l,  s1,  ...,sk+l,  ...,sK;m-l) 


I K 


min{A(0)  + Z B(j)s  +a  Z Z P .,p(i,)  R ,...,s  5m-l) 

j=l  J i'=0  .1=1  U1  J a 1 K 


C(0 


K 


,1) +B(l)  + Z B(  j ) Sj  + aR^(o,  1, s^+1,  s^, . . . , s^; m- 


1)} 


ik 

> P + Z B(J)sj  + B(k)  + QtRa(0,l, s1, . . .,sk+l, . . .,sK;m- 

i) 


1) 


- (C(0, 1)  + B(  1)  + Zb(J)s,  + aRa(0,l,sL+l,s2,...,sK;m-l)) 

> (P  - C(0,1))  + (B(k)  - B(l) ) by  inductive  assumption  on  (a)  for  n = m-l 

> 0,  from  1 and  3. 

Hence,  for  (i,k' , s1#  . . . , sK)  € $ x /S_1  U J ®,  (b)  holds  for  n = m, 
completing  the  mathematical  induction  and  yielding  that  both  assertions 
(a)  and  (b)  hold  for  n > 1.  □ 
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• '«k'  ’ 


( 


Lemma  2.2.  Suppose  conditions  1,  2 and  5 of  Lemma  2.1  hold.  Furthermore, 
assume 

4.  C(i,k)  is  nondecreasing  in  k (1  < k < K)  for  each  fixed  i 

(0  < i < D. 

Then  V (i,k,s,, . . . ,s  jn)  is  nondecreasing  in  k (l  < k < K)  for  each 
fixed  (i, s^, . . s^)  £ and  for  n > 1. 


Proof.  Mathematical  induction  is  applied.  For  n = 1,  for  (i,k, s^, . . . , s^) 

cjl  X f(x  JS, 

K K 

V (i,k,s  ,...,s  ;1)  =min{A(i)  + E B(j)s.,C(i,k)+B(k)  + E B(J)s.) 
a 1 K 3=1  J 3=1  J 


is  clearly  nondecreasing  in  k,  from  1 and  4.  Suppose  the  assertion 
holds  for  n = m-1  > 1,  and  consider  the  case  for  n = m.  The  first  term 
of  the  right  hand  side  of  V (i,  k,  s.. , . . . , s„;  m)  for  (i,k,  s.. , . . . , s„) 

Cl  1 A 11V 

zJl  X /(x  Js  is  independent  of  k,  so  it  is  enough  to  check  the  second 
term.  For  ( i,  s^, ...»  s^)  (L  ^ x £ , and  for  1 < k < K-l, 

<Va(i,k+l,s1,...,sK,m))2<>nd  - 


K 

- C(i,k+1)  +B(k+1)  + E B(j)s  +a«  (0,l,s  , 

j=l  J a 

K 

- (C(i,k)  *B(k)  + E B(j)s  +aRa(0,l,Sl,. 
J 


»sk+1+l, . . .,sKjm-l) 
s^+1 1 . . . > s^;m-l) ) 


> 0,  from  1,  4 and  the  result  (a)  of  Lemma  2.1. 

Hence  the  assertion  holds  for  n = m,  completing  the  mathematical  induction 
and  the  proof  of  the  lemma.  □ 
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Le^a  ?,?,  Suppose  a function  V(i,k)  is  nondecreasing  in  i (0<i<I) 

for  each  fixed  k (l  < k < K)  and  nondecreasing  in  k (l  < k < K)  for 

each  fixed  i (0  < i < I),  and  suppose 

5 P (•)  c P (*)  for  0 < i < 1-1  where  P.  (j)  = E 

) - ri+1v  - - i i.<j  11 

6.  P^^(')  for  0 < i < 1-1,  where  P(i)(k)  = E 

k'<  k 


1 K /j  i \ 

, E E P , P , 'V(i',k')  is  nondecreasing  in  i for  0 < i < I. 


Then, 


i’=0  k'=l 


'ii'  k 


Proof.  For  0 < i'  < 1-1, 

E p^'+1)  v(i'+i,k’)  > E p^’+1)  V(i',k'), 
k'=l  k'=l 

since  V(i',k')  is  nondecreasing  in  i’  for  fixed  k' 

K fi-'l 

> E p^7  ] V(i’,k’)  , 
k’=l 


by  6 and  since  V(i’,k')  is  nondecreasing  in  k' 


which  gives  that  g(i')  = E£,=;l  P^.^' } V(i',k')  is  nondecreasing  in  i' 
(0  < i'  < I).  From  5,  this  implies  that  E*I=0  Pu,  g(i')  is  non- 
decreasing in  i (0  < i < I),  which  is  what  we  want.  □ 


Lemma  2.U.  Assume  conditions  1 though  6 of  the  last  three  lemmas 
hold,  and  furthermore  suppose 

7.  C(i,k)  is  nondecreasing  in  i (0  < i < I)  for  each  fixed  k 

(1  < k < K). 
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ft.  A(i)  is  nondecreasing  in  i for  0 < i < I. 

Then  V ( i, k, s , . . . , s \ n)  is  nondecreasing  in  i (0  < i < I)  for 
Oi  1 — — 

each  fixed  (k, s^, . . . , s^)  € t(  x ^ and  for  n > 1. 


Proof.  Mathematical  induction  is  again  used.  For  n = 1,  for 

it  is  obvious  that  V (i,k, s^, . . . , s^; 1)  is 

nondecreasing  in  i from  7 and  8.  Suppose  the  assertion  is  true 
for  n = m-1  > 1,  and  consider  the  case  for  n ~ m.  Then  for 
(k, s^, . . • > s^)  X ^ , 


i»k, s^, 


• f 1ft) 


K IK 

= min(A(i)  + ZB(j)s.+a  E Z 
j=l  J i ' =0  k' =1 


P^lPj^r  > s^>  • • • > Sj^J  Ift-1  ) t 


K 

C(i,k)+B(k)+  E B(j)s  +OR  (0,1,  s , 
j=l  J 1 


•’Sk+1' 


.,sK;m-l)}  . 


The  second  term  is  obviously  nondecreasing  in  i from  7,  and  as 

V (i,k, s.. , . . . , s >m-l)  is  nondecreasing  in  i by  the  inductive  assump- 

CL  -L  lv 

tion,  and  also  is  nondecreasing  in  k by  Lemma  2.2,  so  is 
RQ(i,k, s^, . . . , sK;m-l)  by  definition.  Using  Lemma  2.5  indicates 
that  ^'i'=0  ^k'=i  pii'pk'  Ra(i'»k'*s1»...»sK;m-l)  is  nondecreasing 
in  i (0  < i < I).  As  A ( i ) is  nondecreasing  in  i (0  < i < I) 
from  8,  so  is  the  first  term  of  V (i,k, s, , . . . , svJm) , yielding  that 

V (l,k,s^, . . . ,sKjm)  is  nondecreasing  in  i for  0 < i < I,  which 

completes  the  mathematical  induction  and  the  proof.  □ 


18 


Now  using  the  above  lemmas,  we  can  prove  the  following  main  theorem. 


Theorem  2.5.  Assume  the  following  conditions  hold: 

1.  B(k)  is  nonnegative  and  nondecreasing  in  k for  1 < k < K. 

2.  C(i,k)  is  nondecreasing  in  k (l  < k < K)  for  each  fixed  i 

(0  < 1 < I). 

3-  C(i,k)  is  nondecreasing  in  i (0  < i < I)  for  each  fixed  k 

(1  < k < K). 

4.  A(i)  - C(i,k)  is  nondecreasing  in  i (0  < i < I)  for  each 
fixed  k (1  < k < K). 

5.  P>C(0,1). 

6.  \ > <3k+1  > 0 for  1 < k < K-l. 

7.  P±(* ) C Pi+1(,)  for  0 - 1 ^ I_1* 

8.  P(i^(-)  C P^i+L^(-)  for  0 < i < 1-1. 

Then  there  exists  a stationary  i control  limit  policy  which  minimizes 
the  total  expected  a-discounted  cost  of  the  basic  multi-repair-type 
maintenance  model. 

Proof.  First  notice  that  if  all  the  conditions  are  satisfied.  Lemma  2.4 
holds  since  conditions  3 and  4 of  this  theorem  imply  condition  8 of 
Lemma  2.4  and  all  other  conditions  are  exactly  the  same. 

For  ( i,  k,  s^, . . *»sK)  £jl  x /{ x /JS  and  for  n > 0,  let 

f n+1 ( , s^) 

= <va(i’^V‘--'sK,n+1)>l-st  " (Va(i'k’si"--’sK;n+1)>2-nd 
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' ••  

* . . _ . J 


ft.  \ ft.  f i ' ' . 

= A(i)  + E B(j)s.  + a E E p . . , Pv • ' R^(l',k',s  jn) 

3=1  j i'  =0  k'  =1  11  k a 1 K 

K 

• (C(  i,k  + B(k1  + . . 1 , s^, . . . , s^+1, . . . , s^;  n j j . 

j ~1 

Now  by  Lemmas  2.2  and  2.^,  V ( i ' , k' , s , . . . , s ; n)  n > 0)  is  both 

ct  l iv  — 

nondecreasing  in  i'  (0  < i'  < I)  for  fixed  k' , and  nondecreasing 

in  k'  (1  < k'  < K)  for  fixed  i' . So  is  R ( i ' ,k' , s, , . . . , sv; n) 

(n  >0),  by  definition.  Therefore,  using  Lemma  2.5,  we  have  that 

^i'=0  ^k'=l  Pii,Pk'  ^ Ra(i’>k',s1,...,sK;n)  (n  > 0)  is  nondecreasing 

in  i (0  < i < I).  The  only  other  expression  containing  i is 

A(i)  - C(i,k),  which  is  assumed  to  be  nondecreasing  in  i (0  < i < I), 

by  V.  Hence  for  n > 1,  f (i, k, s , . . . , s ) is  nondecreasing  in  i 

— n l k. 

(0  < i < I)  for  each  fixed  (k, s)  = (k, s^, . . . , s^)  £ x ^ S.  That 

means,  at  the  beginning  of  each  n (n  > 1)  period  problem,  for  each 

(k,s)  x JS,  there  exists  an  s n such  that  (Va(i,k,  s^  . . . , skj  n)  )2_nd 

is  smaller  than  or  equal  to  (Vr  ( i, k, s , . . . , s j n) ) , i.e.,  to  repair 

an  operating  machine  is  optimal  if  and  only  if  i > i . Thus,  the 

k f s f n 

existence  of  an  i control  limit  policy  optimizing  a finite  horizon 
problem  is  guaranteed.  Now  using  the  fundamental  results  of  Markov 
decision  theory,  the  existence  of  a stationary  i control  limit  policy 
minimizing  the  infinite  horizon  problem  can  be  easily  obtained  (see 
Theorem  2.2  of  [5]).  □ 

Interpretation  of  each  condition  is  given  now.  Conditions  1, 

2 and  6 characterize  the  fact  that  the  types  of  repair  works  are  arranged 
so  that  type  1 is  the  easiest  and  type  K is  the  hardest,  since  they  say 
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that  the  labor  cost,  the  material  cost,  and  the  expected  length  of  repair 
time,  all  increases  as  the  type  number  of  repair  work  increases.  Condition 
3 states  that  the  worse  the  machine  is,  the  more  expensive  its  material 
cost  is.  Condition  U requires  that  the  operating  cost  must  increase  more 
than  the  increase  in  material  cost  for  each  type  of  repair.  Condition  5 
gives  a simple  lower  bound  on  the  penalty  cost.  Condition  7 is  the  so- 
called  IFR  property  of  a deteriorating  system.  Lastly  condition  8 
indicates  that  the  worse  the  state  of  the  machine,  the  harder  it  is  to 
repair  (stochastically).  All  of  them  seem  reasonable,  and  none  of  them 
is  seriously  restrictive. 

Suppose  all  the  conditions  of  Theorem  2.5  hold.  Then  for  each 
(k,s)  £ t(  x ^ ’,  there  exists  an  i^  g such  that  for  all  (i,k, s)  with 
i < i to  keep  a machine  in  operation  is  optimal,  and  for  all  (i,k,s) 

K y S 

with  /.  > i,  to  repair  it  is  optimal  at  any  period.  If,  in  addition, 

tXfS 

we  can  show  that  the  optimal  policy  obtained  is  also  a k control 
limit  policy,  then  i becomes  nondecreasing  in  k (1  < k < K)  for 

K,  S 

each  fixed  s t Then  one  reali- 

operating  condition 


012  •••  1-2  1-1  I 

t 1 

y 2 

I repair 

p . 

1 

e ' 

not  1 

K 

repair  

zation  of  an  optimal  policy  for  each 
fixed  s£  is  as  shown  in  Fig.  2.2  . 

The  above  argument  is  justified 
by  the  following  theorem. 


Figure  2.2.  An  Example  of  an 
Optimal  Policy. 
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Theorem  2.6.  If  conditions  1,  2,  5 and  6 of  Theorem  2.5  hold,  then 


there  exists  a stationary  k control  limit  policy  minimizing  the  total 
expected  a-discounted  cost  of  the  melt i -repair-type  maintenance  model. 


Proof.  Just  as  in  the  previous  theorem,  it  is  enough  to  check  that 

f (i,k, s , . . . , s ) is  nonincreasing  in  k (1  < k < K)  for  each 
n i k.  — 

(i,sr  ..  .,s  ) € x J ° and  n > 1.  Now  if  conditions  1,  5 and  6 

of  Theorem  2.5  hold,  then  Lemma  2.1  holds  and  hence, 

P (0,l,s^, . . . ,s^+l, . . . , s^jn)  (n  > 1)  is  nondecreasing  in  k 

(1  < k < K).  From  conditions  1 and  2 of  Theorem  2.5,  both  B(k)  and 

C(i,k)  are  nondecreasing  in  k (l  < k < K).  As  all  other  terms  do 

not  contain  k in  their  expressions,  we  have  f (i,k, s ,...,sv)  is 

n 1 K 

nonincreasing  in  k (1  < k < K) . □ 


The  long-run  expected  average  cost  version  of  Theorems  2.5 
and  2.6  can  be  easily  obtained  and  stated  as  below: 


Theorem  2.7.  Suppose  all  the:  conditions  in  Theorem  2.5  (2.6)  hold, 
and  furthermore  suppose  that  any  operating  machine  eventually  fails. 
Then  there  exists  a stationary  i control  limit  policy  (k  control  limit 
policy,  respectively!  minimizing  the  long-run  expected  average  cost  of 
the  basic  multi -repair-type  maintenance  model. 

Proof.  Since  anj  operating  machine  eventually  fails,  and  a failed 

machine  must  be  repaired  at  once,  and  q > 0 for  any  k (1  < k < K), 

""" 

(i, k, s^, . . . , s^)  = (0, 1,0, . . . ,0)  is  accessible  from  every  other  state 
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no  matter  what  stationary  policy  is  employed.  Hence,  the  existence  of 
a stationary  policy  optimizing  the  problem  is  guaranteed.  The  rest  of 
the  proof  ‘ s Just  the  same  as  that  of  Theorem  2.3  of  [5]  and  hence  can 
be  omitted.  □ 

In  the  aforementioned  model,  we  assumed  that  the  type  of 
repair  work  required  was  knov/n  to  the  decision  maker  before  he  made  a 
decision.  Now  we  briefly  discuss  the  case  where  the  type  of  repair  work 
required  is  not  known  to  the  decision  maker  In  this  modified  model,  if 
an  operating  machine  in  the  i-th  operating  condition  is  chosen  to  be 
repaired,  it  is  randomly  sent  to  the  type  k repair  shop  with  probability 
p,  and  the  decision  maker  has  no  knowledge  of  where  it  is  to  be  sent 
before  he  makes  a decision.  We  can  define  an  i control  limit  policy  for 
this  model  as  in  the  previous  model,  and  the  sufficient  conditions  under 
which  an  optimal  policy  is  of  an  i control  limit  form  are  to  be  studied. 
Though  the  model  description  is  not  much  different  from  the  basic  model, 
the  analysis  becomes  mucn  more  complicated,  and  some  bounding  technique 
on  the  value  of  the  cost  criterion  must  be  applied  to  derive  sufficient 
conditions  (see  [5]).  The  main  difference  in  the  results  is  in  the 
condition  which  relates  the  operating  cost  and  other  repair  costs.  The 
following  inequality  replaces  condition  4 of  Theorem  2.5: 

K K 

A(i+1)-A(i)>  £ P;i+1)(C(i+l,k)  +B(k))  - £ p£i}(C(i,k)  +B(k)) 

“k=l  k=l 

K 

+ £ max(p‘1+1)  - p^.O)  (--  - 1)  (P  + B(k) 
k=l  K K qk 

K (a) 

- minfA(O),  £ p£ ’ C(0, k)  + B(l)})  , 0 < i < 1-1. 

k»l  K 
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The  third  term  of  the  rignt  hand  side  of  the  above  expression 


gives  the  effect  of  the  penalty  cost,  and  its  value  can  be  comparatively 
large.  But  if  we  further  assume  that  the  repair  time  distribution  is 
independent  of  the  type  of  repair  work,  then  the  penalty  term  vanishes, 
and  the  property  that  A(i)  - 0^  ^ p^  '(C(i,k)  + B(k  /q))  is  non- 
decreasing in  i (0  < i < I)  can  be  shown  to  be  sufficient.  Other 
conditions  are  essentially  the  same  as  those  in  Theorem  2.5. 


2b 


2.2.  Extension 


I 


In  this  section  we  generalize  the  basic  multi- repair-type 
maintenance  model  in  the  following  manner  The  operating  cost  of  a 
machine  depends  on  the  type  of  repair  work  assuming  it  is  repaired, 
as  well  as  its  operating  condition.  Also,  tne  type  of  repair  work 
on  a machine  to  be  repaired  depends  on  the  type  of  repair  work  which 
has  been  required  on  the  machine  if  the  repair  decision  had  been  chosen 
one  period  earlier.  Define 

A(i,k):  operating  cost  for  a machine  in  the  i-th  operating  condition 

which  requires  type  k repair  work  if  it  is  repaired. 

< i * jji 

p,  , ’ « probability  that  a machine  in  the  i'-th  operating  condition 

requires  type  k'  repair  if  the  repair  decision  is  chosen  given 
that  the  type  k of  repair  has  been  required  one  period  earlier. 
This  generalization  seems  to  be  reasonable  when  the  previous 
example  in  Section  2.1  is  considered  There,  the  age  of  a machine  and 
its  degree  of  deterioration  correspond  to  the  operating  condition  of 
a machine,  and  the  type  of  repair  work  on  it  if  repaired,  respectively. 

It  is  reasonable  to  view  the  operating  cost  as  depending  on  the  degree 
of  deterioration,  and  the  degree  of  deterioration  of  a machine  as  depending 
on  the  degree  of  deterioration  one  period  earlier 

Under  this  generalization,  the  following  theorem  holds! 

Theorem  2.8.  Assume  the  following  conditions  hold; 

1.  B(k)  is  nonnegative  and  nondecreasing  in  k for  1 < k < K. 

2.  C(i,k)  is  nondecreasing  in  k (l  < k < K)  for  each  fixed  i 

(0  < i < I). 
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3.  C(i,k)  is  nondecreasing  in  i (0  < i < I)  for  each  fixed  k 

(1  < k < K). 

U.  A(i,k)  is  nondecreasing  in  k (l  ^ k < K)  for  each  fixed  i 

(0  < i < I). 


5. 

A(i,k)  - C(i,k) 

is  nondecreasing  in 

i (0  < i < I)  for  each 

fixed  k (1  < k 

< K). 

6. 

P > C(0,1). 

7. 

\ > Vl  > 0 f0r  1 ^ k < K-l. 

8. 

V>  c pm(-> 

for  0 < i < 1-1. 

9. 

p(iM»(.)  cp(i' 

»k+1)(.)  for  i < k 

< K-l,  0 < i'  <1,  where 

P(l',k'(k’)  = Z 

A <k’ 

10. 


p(i',k)(.)  Cp(i,+1»k'(.)  for  o < i'  < 1-1,  1 < k < K. 


Then  there  exists  a stationary  i control  limit  policy  which  minimizes 
the  total  expected  a-discounted  cost  of  this  generalized  multi-repair- 
type  maintenance  model.  For  the  long-run  expected  average  cost  criterion, 
the  eventual  failure  property  on  any  machine  is  also  needed. 


Proof.  The  dynamic  programming  formulation  of  this  generalized  model 

is  exactly  the  same  as  (2.1)  if  A(i,k)  and  P^,  ’ ' replace  A(i)  and 
( i • ) 

p^,  respectively.  Lemma  2.1  still  holds  after  these  replacements, 
and  Lemma  2.2  also  holds  if  conditions  k and  9 are  added  since  they  assure 
that  (Va(i,k,s1,...,8K;n))_  t is  nondecreasing  in  k in  the  proof. 

It  is  clear  that  Lemma  2.3  holds  if  condition  6 of  Lemma  2.3  is  replaced 


?6 


by  condition  10  of  Theorem  2.8.  As  in  the  proof  of  Lemma  2.4,  we  can 

show  that  V ( i, k, s. , . . . , s ; n)  is  nondecreasing  in  i (0  < i < I for 
CK  i Iv 

each  fixed  (k,  s^, . . . , sR)  C \ and  for  n > 1,  if  conditions  1 

through  7 and  9 and  10  hold,  and  if  A(i,k;  is  nondecreasing  in  i 
(0  < i < I)  for  each  k (l  < k < K) . Finally  it  can  be  shown  that 
f (i,k, s1>...,sK)  is  nondecreasing  in  i for  (k, s^ . . . , sK)  € x , 

and  for  n > 1 if  all  the  conditions  hold,  which  yields  the  desired 
results.  □ 

Condition  9 says  that  the  heavier  the  current  repair  work 
required  on  a machine  is,  the  heavier  that  its  repair  work  will  be  in 
the  future.  Condition  10  means  that  the  worse  condition  a machine  is 
in,  the  more  repair  work  required  on  it  if  it  is  repaired.  None  of 
the  conditions  in  Theorem  2.8  are  very  restrictive  conditions. 
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2.3.  One  Repairman  for  Each  Repair  Shop  Case 


We  return  to  the  basic  multi -repair-type  maintenance  model. 

In  this  model,  the  repair  work  begins  immediately  on  any  machine  sent 
to  any  type  of  repair  shop.  This  is  equivalent  to  saying  that  each 
repair  shop  has  more  than  enough  number  of  repairmen.  In  this  section, 
we  consider  the  other  extreme  case,  where  there  is  only  one  repairman 
in  each  type  of  repair  facility.  In  this  case,  if  a machine  is  to  be 
repaired  at  some  type  of  repair  facility,  it  must  wait  until  all  the 
machines  which  have  arrived  earlier  at  the  same  type  of  repair  facility 
are  completely  repaired.  Machines  waiting  for  repair  form  a queue  in 
front  of  each  type  of  repair  facility.  The  system  is  schematically 
shown  in  Figure  2.3. 


queues  repair 
shops 


Figure  2.3.  A Multi -Repair-Type 

System  with  One  Repair- 
man for  Each  Repair 
Facility. 


Let  ^ (1  < k < K) 
be  the  probability  that  a 
machine  in  type  k repair 
shop  at  the  beginning  of  a 
period  is  completely  re- 
paired at  the  end  of  the 

period.  Let  q^j  be  the 
s s 

probability  that  s'  machines 
are  still  in  the  type  j 
repair  system  at  the  end  of 


the  period  given  that  s 

machines  are  in  the  type  i repair  system  at  the  beginning  of  a period. 
Here  the  type  k repair  system  includes  the  type  k repair  shop  and  the 
queue  formed  in  front  of  the  type  k repair  shop.  Then 
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= 0. 

The  dynamic  programming  formulation  of  this  model  is  the  same 

as  (2.1)  where  q^.'s  are  not  binomial  distributions  as  before  but 
s . s . 
j i 

are  given  as  above.  As  we  can  see  from  the  proofs  of  Lemma  2.1  through 

Theorem  2.5,  q^\'s  explicitly  appear  only  in  the  proof  of  Lemma  2.1. 
s . s . 

J J 

Hence,  if  Lemma  2.1  can  be  proved  under  this  modified  model,  we  can 
obtain  the  sufficient  conditions  for  the  optimality  of  a stationary  i 
control  limit  policy  by  using  the  result  of  Theorem  2.5.  Now  in  the 
proof  of  Lemma  2.1,  it  is  sufficient  to  show  that  for  (i, k' , s, , . . . , s„) 
€<>?  x /^x  ^ S_1  u^q  and  n > 1, 


Va( i>  k . . . , n ) > Va( i > R , • • • * . . . jS^jn) 


implies 

R^(  i,k  f s^f  • , . f s ^ R^  v ^ ; s^lf . , . ^ ^ 9 

and 

Va(  i#  k * • • • t • • • 9 n)  ^ V^(  i > k'  > • • • > ® * * • 9 

implies 

R^(ijk  f • « • , s^*^l|  » • • , Sj^j  n)  > R^(i,k’,  Sx,  • • • 9 • • • 9 jn), 

since  the  other  arguments  do  not  contain  q^\'s  explicitly.  The 

SjSj 
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defined 


(1) 

definition  of  R and  the  simple  substitution  of  q'  , 

a s .s' 

above  give  the  desired  result.  As  a conclusion  of  this  section, 
we  have  Theorem  2.9. 

Theorem  2.7*.  If  all  the  conditions  in  Theorem  2.5  hold,  then  there 
exists  a stationary  i control  limit  policy  which  minimizes  the  total 
expected  a-discountea  cost  of  the  multi-repair-type  maintenance  model 
with  one  repairman  for  each  repair  shop.  For  the  long-run  expected 
average  cost  criterion,  the  eventual  failure  property  on  any  machine 
is  also  needed. 





CHAT  TER  3 


MAINTENANCE  MODEL 3 WITH  CONTROL  OF  QUEUE 

In  this  chapter,  maintenance  models  are  treated  in  the  context 
of  control  of  queues.  In  these  models,  the  decision  maker  ha.:  the  option 
of  opening  or  ciosing  the  repair  shop  when  there  are  machines  waiting 
for  repair  service,  as  well  as  the  option  of  repairing  or  leaving  an 
operating  machine  alone.  A two-dimensional  control  limit  policy  is 
defined,  and  sufficient  condition,  for  the  optimality  of  a two-dimensional 
control  limit  policy  are  obtained  for  each  model.  Lastly,  some  computa- 
tional remarks  and  conclusions  are  stated. 

3.1.  The  Model  and  Some  Example  • 

We  consider  the  following  discrete  time  Markov  maintenance  model, 
whose  mechanism  is  illustrated  in  Fig.  3.1.  There  is  an  operating 

machine  and  S (S  > 1) 
identical  spare  machines  in 
the  system.  At  the  beginning 
of  each  period,  an  operating 
machine  is  classified  as 
being  in  one  of  1+1  (I  > 1) 
states  as  was  done  in  the 
previous  models.  There  is 


operating 

spare 

machine 

’ 

units 

or  leave 


— HXED — h 


option  of  open 
or  close 


queue 


repair 

facility 


J 


Figure  3. 1. 


A Machine  Maintenance 

System  with  Control 
of  Queue. 
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a repair  shop  in  the  system,  and  an  operating  machine  can  be  sent  to  the 
repair  shop  for  the  repair  work  at  any  period.  We  assume  that  there  is 
only  one  repairman  in  the  repair  shop,  or  only  one  machine  can  be 
repaired  in  the  repair  shop  during  any  time  period.  Therefore  a machine 
sent  to  the  repair  shop  must  wait  until  all  the  machines  which  have 
already  arrived  at  the  repair  shop  are  completely  repaired.  Waiting 
machines  form  a queue.  No  waiting  is  necessary  if  there  has  been  no 
machine  in  the  repair  shop. 

At  the  beginning  of  each  period,  just  after  the  observation  of 
the  state  of  the  system,  the  decision  maker  has  the  option  of  opening  or 
closing  the  repair  shop,  as  well  as  the  option  of  repairing  or  leaving 
an  operating  machine  alone.  Therefore  at  each  period,  four  actions  are 
available.  They  are  denoted  as  a^,  a^,  aRC,  and  aRQ  respectively, 
where  L,  R,  C and  0 mean  to  leave  an  operating  machine  in  operation, 
to  repair  an  operating  machine,  to  close  the  repair  service  gate,  and 
to  open  the  repair  service  gate  respectively.  For  example,  a^c  is  the 
action:  leave  an  operating  machine  in  operation  and  close  the  service 
gate.  If  there  is  no  operating  machine,  or  if  all  the  machines  are  in 
the  repair  system  consisting  of  a repair  shop  and  a queue,  the  option 
of  repairing  or  leaving  an  operating  machine  is  not  available.  Only 
the  option  of  opening  (a^)  or  closing  (ac)  the  service  gate  is 
then  available.  Closing  the  repair  service  gate  means  closing  it  if  it 
has  been  open  and  keeping  it  closed  if  it  has  been  closed.  Similarly, 
opening  the  repair  service  gate  implies  opening  it  if  it  has  been  closed 
and  keeping  it  open  if  it  has  been  open.  The  repair  work  can  be  performed 
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only  when  the  service  gate  is  open.  If  a repair  job  on  a machine  is 
interrupted  by  the  decision  to  close  the  gate,  the  rest  of  the  repair 
work  is  postponed  until  the  gate  reopens.  Repaired  machines  are  avail- 
able as  spare  units. 

If  the  decision  to  leave  an  operating  machine  in  operation  is 
chosen,  it  keeps  operating  and  its  state  evolves  from  i to  j in 
one  period  according  to  the  transition  probability  p.  . >0.  If  the 
decision  to  repair  an  operating  machine  is  selected,  it  is  immediately 
sent  to  the  repair  system,  and  is  instantly  replaced  by  a spare  unit, 
if  any  are  available.  The  new  operating  machine  begins  to  operate  just 
after  replacement  in  its  best  condition. 

The  costs  associated  with  the  system  are: 


A(i) 


C(i) 


K(s,k) 


E 

F 

G 

P 


cost, 


: operating  cost  for  a machine  in  the  i-th  operating  condition 
(0  < i < I)  per  period. 

: material  cost  for  repairing  a machine  in  the  i-th  operating 
condition  (0  < i < I). 

: holding  cost  per  period  of  s (0  < s < S+l)  machines  in 
the  repair  system  when  the  gate  is  closed  (k  = 0)  or  open 
(k  1)  at  the  beginning  of  the  period. 

: set  up  cost  of  opening  a closed  repair  shop. 

: shut  down  cost  of  closing  an  open  repair  shop. 

: service  cost  of  operating  an  open  repair  shop  per  period. 

: penalty  cost  assessed  per  period  while  no  operating  machine 
is  available. 

The  objective  function  is  the  total  expected  a-discounted 
and  the  structure  of  an  optimal  policy  minimizing  such  a criterion 


is  studied. 
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Before  proceeding  further,  we  give  some  examples  to  clarify 


the  applicability  of  this  model  in  the  practical  world. 

The  first  example  is  an  airplane  repair  problem  for  a privately 
owned  flying  shcool.  Suppose  the  instructor  in  the  flying  school  owns 
two  airplanes  and  a repair  shop  for  repairing  them.  He  teaches  flying 
using  one  airplane  at  a time.  He  inspects  the  condition  of  the  airplane 
in  service  periodically,  and  he  classifies  it  as  being  in  one  of  a 
finite  number  of  states.  He  continues  using  the  same  airplane,  until 
he  judges  that  it  should  be  repaired.  Then  the  other  airplane,  if 
available,  replaces  it  and  begins  to  operate  in  its  best  condition. 

The  previously  used  airplane  is  sent  to  the  repair  shop,  which  is 
either  open  or  closed.  Repair  work  can  be  performed  only  when  the 

repair  shop  is  open,  and  a repaired  airplane  will  be  ready  for  future 

use  in  its  best  condition.  Keeping  the  repair  shop  always  open  may 

not  be  economical  since  he  must  pay  salary  to  a repairman,  and  other 

costs  to  keep  it  open,  even  when  no  repair  work  is  needed.  Keeping 
the  repair  shop  closed  too  long  is  also  undesirable  since  it  may 
prevent  airplanes  from  being  available,  thereby  resulting  in  a loss  of 
revenue.  The  problem  of  finding  the  best  schedule  for  hiring  a repair- 
man and  for  replacing  an  operating  airplane  can  be  formulated  as  a model 
to  be  studied  in  this  chapter. 

Another  example  pertains  to  cleaning  suits.  Suppose  a person 
has  several  suits.  He  wears  one  suit  continuously  until  he  finds  that 
it  requires  cleaning,  and  then  he  changes  the  suits  using  one  remaining 
in  his  wardrobe.  Assume  the  degree  of  cleanliness  of  a suit  is 

3U 


observable.  The  cleaner  the  suit  he  wears  the  better  the  impression 
he  gives,  and  hence,  the  more  comfortable  he  himself  becomes.  In 
this  sense,  the  cost  of  wearing  a suit  is  associated  with  its  cleanli- 
ness. When  he  decides  to  change  the  suit  to  the  clean  one,  it  does 
not  necessarily  follow  that  he  immediately  sends  the  used  suit  for 
dry-cleaning  since  such  an  action  may  be  laborious  and  time  consuming. 
However,  too  infrequent  visits  to  the  cleaners  may  lead  to  the  situation 
where  he  has  no  clean  suits  available.  Assuming  that  a suit  is  avail- 
able in  its  cleanest  condition  after  dry-cleaning,  the  problem  of 
determining  the  best  schedule  for  suit  changes  and  laundry  visits 
can  be  formulated  as  a model  to  be  studied. 

3.2.  Control  Limit  Policy  with  Respect  to  Operating  Machine 

In  the  last  section,  we  did  not  specify  the  repair  time  distri- 
bution. It  is  clear,  though,  if  the  repair  service  gate  is  closed,  no 
repair  work  can  be  performed.  When  the  repair  service  gate  is  open, 
we  assume  the  following  in  this  section;  The  number  of  machines  in 
the  repair  system  solely  determines  the  distribution  of  the  number  of 
machines  in  the  repair  system  one  period  later.  Let  q , be  the 
probability  that  s'  machines  are  still  in  the  repair  system  at  the 
end  of  the  period,  given  that  s machines  are  in  the  repair  system 
at  the  beginning  of  a period. 

In  order  to  describe  the  model,  we  need  to  specify  the  number 
of  machines  in  the  repair  system  and  the  state  of  the  repair  service 
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gate,  as  well  as  the  condition  of  an  operating  machine  Let  VQ,(i,s;n) 

be  the  minimum  expected  n period  a-discounted  cost  given  that  the 

operating  machine  is  in  the  i-th  operating  condition,  the  number  of 

machines  in  the  repair  system  is  s,  and  the  state  of  the  repair 

service  gate  is  k i,k  0 means  the  gate  is  closed,  and  k 1 means 

it  is  open'  at  the  beginning.  Then  by  letting  V^(i,s;n)  = 0 for 

It 

any  feasible  i,  s,  and  k,  V^(i,s;n)  (n  > 1)  satisfies  a set  of 
recursive  equations: 

For  0 < i < I,  0 < s < S, 


A(i)  +K(s,0)+a  Z p V (j,s;n-l), 

0=0  a 

1 3 1 

A(i)  + K(s,0)  + E + G + a Z p.  . Z q , vU j, s 1 , n- 1) , 

0=0  1J  s ’ =0  ss  a 


C(i)  + K(s,0)  + a Z P0.  V ( j, s+1; n-1) , 

0=0  J 

I s+1 

C(i)  + K(s,0)  + E + G + a Z p Z q . V„( j, s 1 ; n-1) 

0=0  J s’=0  ’ 


= min 


A ( i ) + K(s,l)  + F + a Z P,,V^(j,s;n-l), 

0=0  a 

1 s 1 

A(i)  + k(s,i)  + g + a Z P Z q„Q,v^(d, s' ;n-l), 

0=0  s' =0  ss  a 


C(i)  + K(s,l)  + F + a Z p V^(  j, s+1; n-1) , 

j=0  J a 

I s+1  , 

c( 1)  + K(s,l)  + G + a Z p Z q V (j, s', n-1) 

0=0  UJ  s'=o  s i,s  a / 


V°(i,S+l5n) 


min 


P + K(S+1,0)  + a Z p V°(j,S+l;n-l), 

UJ  a 

I S+l 

P + K(S+1,0)  + E + G + a I p0,  ),  q , V (j,s';n-l) 

j=0  a'  =0  b i,S  a 


V^(i,S+l;n) 


'p  + K(s+l,l)  + F + a I p_,  V°  ( J , S+l;  n-i ) , 


= min 


j=0 

I 


0j  a' 

S+l 


P + K(S+1, 1)  + G + a L pn,  T Mq+1  . V*(j,s 
^ j=0  0J  8’=0  ’ 


(3.1) 


Note  that  <V^(l,B,n)>x_st,  <v£(i,s;n)>2_nd,  <V*(i, s; n) >3_rd,  and 
(V^(i,  s»n)>it_th  are  the  n Period  costs  of  taking  a^,  a , aRC, 
and  ap  respectively  at  the  beginning  followed  by  the  best  policy. 
When  s = S+l,  a^  and  are  the  only  available  actions.  Also  note 

that  the  i in  V (i,S+l$n)  is  a dummy,  and  is  used  just  for 

U- 

notational  simplicity.  When  all  the  machines  are  in  the  repair  system, 
no  operating  machine  is  available,  and  i in  the  above  expression  has 
no  meaning. 

As  the  system  is  a Markov  decision  process  with  discount 
factor  0 <at  <1,  the  existence  of  a stationary  policy  minimizing  the 
total  a-discounted  cost  is  guaranteed.  For  further  discussion  on  the 
structure  of  an  optimal  policy,  the  following  simplification  is  useful. 
Let 
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(5.2) 


* 0 

R (i,s;n  a 7.  p V ( J>  Sj  n) 
j-0  J 

1 s 1 

Q (i,s;n)  =a  ) p 7.  a , V (j,s',n)  . 
a J=0  J s'  =0  ss  a 

Then  for  n _ 1,  0 < i < I,  0 < s _ S.  (5.1)  can  be  simplified  to 
V0(i,s}n) 

minfA^i)  + K(s,0)  + R (i,s;n-l),  A(i)  + K(s,0)  + E + G 

ot 

+ Qa(i,s;n-1),  C(i)  + K(s,0)  + Rq(0, s+1; n-1) , C(i) 

+ K(s,0)  + E + G + Q (0, s+1; n-1) } , 

01 

V*(i,s;n) 

= min(A(i)  + K(s,l)  + F + R^(i,s;n-1),  A(i)  + K(s,l)  + G 
+ Q^(i, s; n-1) , C(i)  + K(s,l)  + F + R^(0, s+1; n-1) , 

C(i)  + K(s,l)  + G + 0^(0, s+1; n-1)}  , 

V°(i,S+l;n) 

min{P  + K(S+1,0)  + R (0,  S+i;  n-1) , P + K(S+1,0)  + E + G + ^(0,S+lJ  n-1) ) , 
Va(i,S+l;n) 

min{P  + K(S+1, 1)  + F + R (OfS+l»n-l),P  + K(S+l,l)  +G  + Q (0,S+ljn-l))  . 

(5.5) 

The  problem  is  now  to  find  the  structure  of  an  optimal  policy. 

It  is  conceivable  that  an  optimal  policy  has  the  form  that  the  repair 
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decision  is  taken  if  and  only  if  the  condition  of  an  operating  machine 
becomes  worse  than  some  critical  value,  and  that  the  decision  to  open 


the  repair  service  gate  is  taken  if  and  only  if  the  number  of  machines 
in  the  repair  system  exceeds  some  critical  value.  The  following  types 
of  control  limit  policies  will  then  be  suitable  candidates  as  the  class 
of  policies  satisfied  by  an  optimal  policy. 

Definition.  A control  limit  policy  with  respect  to  an  operating  machine 

is  a nonrandomized  policy  where,  as  the  option  of  repairing  or  leaving 

an  operating  machine  is  concerned,  there  is  an  i for  each  k 

(k  = 0,1),  s (0  < s < S),  and  n (n  > 1),  say  i,  , called  the 

K J S y n 

control  limit,  such  that  for  all  (i,k, s)  with  i < i , the  decision 

K f S y n 

| at  period  n is  to  leave  it  in  operation,  and  for  all  (i,k, s)  with 

i > i,  , the  decision  at  period  n is  to  repair  it.  A control  limit 
- k,  s, n 

policy  with  respect  to  a repair  shop  is  a nonrandomized  policy  where, 

as  the  option  of  opening  or  closing  the  repair  service  gate  is  concerned, 

there  is  an  s for  each  k (k  = 0,  1) , i (0  < i < I) , and  n (n  > 1), 

say  s,  , , called  the  control  limit,  such  that  for  all  (i,k, s)  with 
k,  l,  n 

s < s.  . , the  decision  at  period  n is  to  close  it,  and  for  all  (i,k, s) 

k,  l,  n 

with  s > s , , the  decision  at  period  n is  to  open  it.  A two- 

k,  l,  n 

dimensional  control  limit  policy  is  a control  limit  policy  with  respect 
to  both  an  operating  machine  and  a repair  shop. 

The  rest  of  this  section  concentrates  on  finding  sufficient 
conditions  for  the  optimality  of  a control  limit  policy  with  respect  to 
an  operating  machine.  The  following  lemma  holds: 
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Lemma  3.1.  Assume  the  following  conditions  hold; 


I 

1.  A(i)  is  nondecreasing  in  i for  0 < i < I. 

2.  C(i)  is  nondecreasing  in  i for  0 < i < I. 

3.  Pt(0  C P1+L(-)  for  0 < i < 1-1. 

Then  for  each  fixed  s (0  < s < S),  k (k  = 0,1),  and  n > 1,  V^(i,s>n) 
is  nondecreasing  in  i (0  < i < I). 

Proof.  Mathematical  induction  is  applied.  For  n = 1,  for  each  s 

(0  < s < S)  and  k (k  = 0,1),  V^(i,s;l)  is  clearly  nondecreasing 

in  i (0  < i < I),  by  1 and  2.  Suppose  the  assertion  holds  for 

n = m-1  > 1.  Then  by  the  definitions  of  R and  Q , both  R (i,s;m-l) 

OL  OL  OL 

and  (^(i.sjm-l)  become  nondecreasing  in  i (0  < i < I)  because  of  3. 
j Hence,  all  the  terms  on  the  right  hand  sides  of  both  V^(i,s$m)  and 

V^(i,s;m)  are  nondecreasing  in  i (0  < i < I)  if  1 and  2 are  satisfied, 
which  completes  the  mathematical  induction  and  the  proof.  □ 

Next  theorem  assures  the  optimality  of  a stationary  control 
limit  policy  with  respect  to  an  operating  machine. 

Theorem  If  conditions  2 and  3 of  Lemma  3.1  hold,  and  if 

i*.  A ( i ) - C(i)  is  nondecreasing  in  i for  0 < 1 < I, 
then  there  exists  a stationary  control  limit  policy  with  respect  to 
an  operating  machine  which  minimizes  the  total  expected  a-discounted 
cost  of  the  maintenance  model  with  control  of  queue. 


k) 


I 


I 


I 


I 


a 


Proof.  We  first  consider  the  n-stage  problem.  For  n > 1,  0 < i < I, 
0 < s < S,  and  k = 0,1,  let 


L, n^1’ 


mi 


n((V*(l, 


sjn)  > 


1-st’ 


<Vl’3>n)W 


fgin(i.s)  = mln{<V^(i,8in))5_ 


rd’  <V£(1-S'">W 


Then  fT  (i,s)  can  be  interpreted  as  the  minimum  n-stage  a-discounted 
cost  given  that  a machine  is  in  the  i-th  operating  condition,  the  number 
of  machines  in  the  repair  system  is  s,  and  the  state  of  the  repair 
service  gate  is  k at  the  beginning,  and  only  the  decision  to  keep  a 
machine  in  operation  is  allowed  at  the  beginning.  If  only  the  decision 
to  repair  an  operating  machine  is  allowed  at  the  beginning,  we  have 

Now,  for  0 < i < I,  0<s<S,  and  n > 0, 


Wl-'> 


fWi-5) 


= A(i)  + K(s,0)  + min{Ra(i, s;n),  E + G + Qa(i,s;n)} 

- C(i)  - K(s,0)  - min(R  (0,s-i-l;n),  E + G + Q (0,s+l;n))  . 

0 1 Of 


As  conditions  2 and  4 of  Theorem  3.2  imply  condition  1 of  Lemma  3.1, 

Lemma  3.1  holds,  which  shows  that  both  R (i,s;n)  and  Q (i,sjn) 

Ct  Of 

are  nondecreasing  in  i (0  < i < I).  With  this,  and  by  3,  we  have  that 

f?  (i,s)  - f„  (l,s)  is  nondecreasing  in  i (0  < i < I)  for  each  s 
n n,  n — — 

(0  < s < S)  and  n > 1.  Similarly,  it  can  be  easily  shown  that 

„(i,s)  - fi  (i,s)  is  nondecreasing  in  i (0  < i < I)  for  each  s 
uy  n n,  n — — 

4l 


(0  < s ± S)  and  n > 1.  Hence,  there  exists  a set  of  critical  numbers 
i^  (k  = 0, 1,  0 < s < S)  for  each  n > 1 such  that 

K,  S 

fL.n(1’s)  ' f!),n<l,s)  for  "ny  1 C 

and 


fL,n<i’s)  ft,ra”y  ‘-‘L- 


Therefore,  as  far  as  the  option  of  repairing  or  leaving  an  operating 
machine  is  concerned,  at  the  beginning  of  each  n-stage  problem,  if  the 
state  of  the  system  is  (i,s,k),  to  leave  a machine  in  operation  is 
optimal  if  its  operating  condition  i is  less  than  i”  , and  to  repair 

rif  S 

it  is  optimal  if  i is  greater  than  or  equal  to  i”  , which  is  a 

K.  y S 

control  limit  policy  with  respect  to  an  operating  machine.  For  the 
infinite  horizon  problem,  the  technique  of  Theorem  2.2  of  [5]  can  be 
applied  here  to  show  the  optimality  of  a stationary  control  limit 
policy  with  respect  to  an  operating  machine.,  □ 


At  the  end  of  this  section,  we  make  a few  remarks  on  the 
relations  between  optimal  decisions  when  the  repair  gate  is  closed 
and  the  corresponding  optimal  decisions  when  the  gate  is  open.  For 
this  discussion,  we  assume  that  £,  F are  nonnegative. 


Lemma  3 . 3 . When  the  repair  service  gate  is  closed  at  the  beginning 
of  a period,  if  a^  is  optimal  for  some  (i,s),  then  a^Q 

(ap0,  respectively)  is  also  optimal  for  the  same  (i,s)  when  the 


gate  is  open. 
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a^.  In  a similar  fashion,  the  state- 

lim  Q (i,s;n) 

a ’ 

n -» <® 

R (i, s)  = lim  R (i,s;n)  . 

OC  Cl 

n « 

For  a fixed  (i,s),  and  k = 0,  a^  being  better  than  a^ 

implies 

A(i)  + K(sf0)  + E + G + QQ(i,s)  < A(i)  + K(s,0)  + I^(i,s)  . 

Hence, 

A(i)  + K(s,  1)  + G + Qa(i,s)  < AU)  + K(s,l)  + E + G + Qa(i,s) 

< A(i)  + K(s, 1)  + Ra(i,s) 

< A(i)  + K(s, l)  + F + Ra(i,s). 

Thus,  for  (i,s)  and  k = 1,  is  better  than  a^.  Similarly,  the 

statement  that  a^  is  better  than  aRC  for  k = 0 implies  that  aLQ 

is  better  than  aRC  for  k = 1,  and  the  statement  that  a^Q  is  better 

than  a^  for  k = 0 implies  that  a^  is  better  than  a^  for  k = 1, 

yielding  that  for  (i,s),  a^  is  optimal  when  the  gate  is  open. 

Lemma  3,4.  When  the  repair  service  gate  is  open  at  the  beginning  of  a 
period,  if  a^,  (a^)  is  optimal  for  some  (i,s),  then  aRC  (a^, 
respectively)  is  also  optimal  for  the  same  (i,s)  when  the  gate  is  closed. 

Proof.  Similar  to  Lemma  3.3,  and  hence  can  be  omitted.  □ 
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Proof.  We  prove  the  statement  for 
ment  for  a^  can  be  proved. 

Let 

- 


The  same  argument  still  holds  for  the  case  where  there  is  no 
operating  machine.  In  that  case,  if  opening  the  gate  is  optimal  when 
the  gate  is  closed,  then  keeping  it  open  is  optimal  when  the  gate  is 
open.  Conversely,  if  closing  the  gate  is  optimal  when  the  gate  is  open, 
then  keeping  it  closed  is  optimal  when  the  gate  is  closed. 


3.3.  Case  Where  Repair  Time  is  Negligible 

In  this  section  sufficient  conditions  to  ensure  the  existence 
of  a control  limit  policy  with  respect  to  a repair  shop  minimizing  the 
total  Qr-discounted  cost  are  of  interest.  The  following  assumption  is 
made  on  the  repair  time  of  a machine  throughout  this  section.  For 
0 < s < S+l, 

1 if  s’  = 0 

qSS'  \ 

* 0 if  s’  4 0 

1 

The  above  assumption  implies  that  the  repair  time  of  each  machine  is 
negligible  compared  with  the  length  of  a period,  and  hence,  all  the 
machines  brought  into  the  repair  system  can  be  repaired  completely  in 
one  period  when  the  service  gate  is  open.  This  assumption  will  be 
reasonable  If  purchasing  or  ordering  machines  takes  place  instead  of 
repairing  them  when  the  "gate  is  open." 

vJe  first  show  the  following  lemma. 

Lemma  3. . Assume  the  following  conditions  hold: 

1.  C(i)  Is  nondecreasi ng  in  i for  0 < i < I. 

K s,k)  is  nondecreasing  in  s (0  < s < S+l)  for  k =0,1. 


2. 


3.  P > c(i). 


Then  V (i,s;n)  is  nondecreasing  in  s (0  < s < S+l)  for  each 
0 < i < I,  k = 0,1,  and  n > 1. 

Proof.  Proof  is  by  mathematical  induction.  For  n = 1,  0 < i < I, 
and  0 < s < S, 

V°(i,s;l)  = minfA(i)  + K(s,0),  C(i)  + K(s,0)}  , 
which  is  nondecreasing  in  s (0  < s < S)  by  2.  Also 


V°(i,S+l;l)  - V°(i,S;l) 

= P + K(S+1,0)  - (min{A(i),  C (i ) ) + K(S,0)) 

> P - min{A(i),  C(i) } , by  2 

> P - C(i)  >0,  by  1 and  3. 

Hence,  V^(i,s;l)  is  nondecreasing  in  s for  0 < s < S+l.  In  a similar 
manner,  it  is  shown  that  V^(i,s;l)  is  nondecreasing  in  s for 
0 < s <_  S+l.  Suppose  the  hypothesis  holds  for  n = m-1  > 1.  Then  for 

0 < s < S, 


V°(i,s;m) 

“ K(s,0)  + min[A( i)  + R^Ci, s ;m-l),  A(i)  + E + G + Qa(i,s;m-1), 

CCi)  + R (0, s+l ;m-l) , C(l)+E  + G + Q (0, s+l, m-1)}  . 

Of  U 

Now  both  R^(i,s;m-1)  and  R^(0, s+l ;m-l)  are  nondecreasing  in  s 
(0  < s < S)  by  the  induction  hypothesis  and  by  the  definition  of  R^, 
and  Qa(i,s;m-1)  is  constant  in  s since 
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Q (i,sim-l  = a 7.  p V^(j,0;m-1)  . 

a j=0  u 

Also  K(s,0)  Is  nondecreasing  in  s (0  < s < 3),  yielding  that 

VC  (i,s;m')  is  nondecreasing  in  s (0  < s * S).  Also, 

0! 

V°(i,S+l;m)  - V°(i,S;m) 
a ’ a ’ ’ 

> P + K(S+1,0)  + min{R  (0,s*l;m-l),  E + G + Q (0,S+l;m-l) } 

a jl 

- (C(i)  + K(S,0)  +minfRa(0,S+l;m-l),  E + G + Q (0,S+l;m-l) } ) 

= P - C(i)  + K(S+1,0)  - K(S,0)  > 0,  by  1,  2 and  5. 

Thus,  V^(i,s;m)  is  nondecreasing  in  s (0  < s < S+l)  for  each  fixed 
0 < i < I.  Similarly,  we  can  show  that  V^(i,sjm)  is  nondecreasing 
in  s (0  < s < S+l),  completing  the  mathematical  induction  and  the 
proof.  □ 

Using  the  above  lemma  and  the  simplfying  assumption  on  <lss,> 
we  can  prove  the  following  theorem,  which  gives  sufficient  conditions 
for  the  optimality  of  a control  limit  policy  with  respect  to  a 
repair  shop. 

X 

0 

Theorem  3.6.  If  all  the  conditions  in  Lemma  3.5  hold,  then  there 
exists  a stationary  control  limit  policy  with  respect  to  a repair 
shop  which  minimizes  the  total  expected  a-discounted  cost  of  the 
simplified  maintenance  model  with  control  of  queue. 
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Proof.  For  n > 1,  0 < i < I,  0<s<S,  and  k =0,1,  let 


fC,n(i,s)  min{  Va(i’S;n))l-st’  (^(i'S5n)>3.rd} 


f0,  n ij S *in{<V*(i,s;n)>?_nd>  <Va(i,s;n))U-tl 


f„  i,s  can  be  interpreted  as  the  minimum  n-stage  a-discounted 
c,  n 

cost  given  that  the  state  of  the  system  is  (i,s,k)  at  the  beginning, 

and  only  the  decision  to  close  the  repair  shop  is  allowed  at  the 

beginning.  If  only  the  decision  to  open  the  repair  shop  is  allowed 

at  the  beginning,  we  have  f„  (i,s). 

O,  n 

The  proof  of  this  theorem  is  similar  to  that  of  Theorem  5.2, 
and  it  is  sufficient  to  verify  that  f^  (i,s)  - f^  (i,s)  is  non- 
decreasing  in  s (0  < s < S)  for  each  fixed  i (0  < i < I),  k 
k = 0,1),  and  n > 1.  But,  for  n > 0,  0 < i < I,  and  k =0,1, 

Ci11'"  - <n+i(1'8) 

= min{A(i)  + R^(i,s;n),  C(i)  + R (0, s+1; n) } 

- minfA(i)  +E + G + Qa(i,s;n),  C(  i)  + E + G + 0^(0,  s+1;  n) ) , 


and  both  R (i,s;n)  and  R (0, s+1; n)  are  nondecreasing  in  s 

Ji 

(0  < s < S)  by  Lemma  5.5,  and  both  0,^(1,  s>n)  and  Q (0,s+ljn) 

are  constant  in  s by  the  simplifying  assumption  on  q , . Hence, 

s s 

f^  (i,s)  - f^  (i,s)  is  nondecreasing  in  s (0  < s < S)  for 
u y n u,  n ~ “■ 

n > 1 and  0 < i < I.  In  a similar  manner,  f^  (i,s)  - flj  (i,s) 
is  shown  to  be  nondecreasing  in  s (0  < s < S),  which  is  what 
we  need.  □ 


Combining  Theorems  3.2  and  J.6  gives  sufficient  conditions  under 
which  a two-dimensional  control  limit  policy  is  optimal. 

Theorem  3-7  Assume  the  following  conditions  hold: 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I. 

2.  A(i)  - C(i)  is  nondecreasing  in  i for  0 < i < I. 

3.  K(s,k)  is  nondecreasing  in  s (0  < s < S-rl)  for  k = 0,1. 

4.  P > C(I). 

5.  P^* ) C P1+1(* ) for  0 < i < 1-1. 

Then  there  exists  a stationary  two-dimensional  control  limit  policy 
minimizing  the  total  expected  a-discounted  cost  of  the  simplified 
maintenance  model  with  control  of  queue. 


One  realization  of  an  optimal  stationary  two-dimensional  control 
limit  policy  is  illustrated  in  Fig.  3.2,  As  previously  pointed  out  in  Lemmas 
3.3  and  3.4,  the  region  where  a^Q  (a^)  °Ptiina^>  called  the  optimal 
region  of  (a^.,  respectively),  when  the  gate  is  open  covers  the 

optimal  region  of  &LQ  (a^,  respectively)  when  the  gate  is  closed. 


k 0 (closed)  k~l  (open) 


Figure  3.2.  A Typical  Optimal  Two-Dimensional  Control  Limit  Policy 
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Further,  the  optimal  region  of  a^  (a^)  when  the  gate  is  closed  covers 
that  of  a^c  (aR(_,,  resPec,tlvely)  when  the  gate  is  open.  That  also  implies 

that  the  boundary  of  two  optimal  regions  of  a^0  and  a^  for  k = 0 

is  also  a boundary  of  the  corresponding  optimal  regions  for  k = 1,  and 
that  the  boundary  of  two  optimal  regions  of  a^c  and  aRC  for  k = 1 

is  also  a boundary  of  the  corresponding  optimal  regions  for  k = 0. 

Notice  that  for  both  k = 0 and  k = 1 cases,  the  boundary  of 
optimal  regions  of  a^  and  aRC  is  vertical,  and  that  of  a^  and 
a^Q  is  horizontal.  This  can  be  seen  by  comparing  the  appropriate 
terms  in  (3-3).  For  example,  to  verify  that  the  boundary  of  optimal 
regions  of  aR0  and  aRC  is  vertical  for  k = 0,  it  is  enough  to 
see  that  the  direction  of  the  inequality 


C(  1)  +K{s,G)  + ®a(0, s+l;n)  = C(i)  +K(s,0)  +E  + G + Q (0,s+l;n) 


does  not  depend  on  i for  each  fixed  s (0  < s < S). 

In  the  theory  of  control  of  the  service  process  by  turning  on 
and  off  servers,  there  is  a class  of  policies  called  hysteresis  loop 
policies  which  are  optimal  under  a wide  class  of  cost  functions.  The 
form  of  the  policy  is:  provide  no  service  if  the  system  size  is  m 
or  less,  and  when  the  system  size  increases  to  M (M  > m),  turn  the 
server  on  and  continue  serving  until  the  system  size  again  drops  to  m. 

In  this  model  we  notice  that  the  control  limit  on  the  action 
of  closing  (a^  or  a^)  or  opening  (a,Q  or  a^)  the  repair 
service  gate  for  k = 0 is  no  less  than  the  corresponding  control 
limit  for  k = 1,  keeping  the  condition  of  an  operating  machine  fixed. 


Thus,  if  we  keep  the  condition  i of  an  operating  machine  fixed,  an  optimal 

or  aRr,  is  taken) 

if  the  number  of  machines  waiting  for  repair  service  is  nu  or  less,  and 
when  the  number  of  machines  waiting  for  repair  service  increases  to  M. 

(Mj  > iik),  open  the  gate  (a^Q  or  aRQ  is  taken),  and  keep  it  open  until 
the  number  of  machines  to  be  repaired  again  drops  to  nu.  This  is  a 
hysteresis  loop  policy  However,  the  actual  sample  paths  or  trajectories 
are  unlikely  to  be  hysteretic  since  the  operating  condition  of  an 
operating  machine  may  change  as  time  passes 


policy  has  the  following  form:  keep  the  gate  closed  (a^ 


5,4,  Case  Where  Repair  Time  Distribution  is  Geometric 

In  order  to  assure  the  optimality  of  a two-dimensional  control 
limit  policy,  we  assumed  in  the  last  section  that  the  repair  time  of  a 
machine  was  negligibly  small  as  compared  with  the  length  of  a period. 
This  assumption  is  relaxed  in  this  section  at  the  cost  of  optimality 
of  a two-dimensional  control  limit  policy.  Here  we  assume  that  the 
repair  time  of  a machine  when  the  repair  service  gate  is  open  has  a 
geometric  distribution  with  parameter  q Then, 


' q 

if 

s ' 

= s-1,  1 < 

s 

< S+l 

l-q 

if 

s 1 

= s,  1 < s 

< 

S+l 

I..’  - { 

0 

if 

s 

/ S j/  s-1, 

1 

<:  s < S+l,  or  if  0 = s / s 

1 

if 

s 

= s’  = 0 , 

v 


since  at  most  one  machine  can  be  repaired  in  the  repair  shop  at  each 
period  no  matter  how  many  machines  are  in  the  repair  system. 
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Consider  a stationary  control  limit  policy  with  respect  to  an 
operating  machine.  The  existence  of  such  a policy  minimizing  the  total 
u-discounted  cost  is  guaranteed  if  the  conditions  in  Theorem  3-  are 
all  satisfied.  In  the  case  of  a stationary  control  limit  policy  with 
respect  to  a repair  shop,  the  analysis  becomes  much  more  complicated. 

In  some  papers,  a convex  analysis  or  a piecewise  convex  analysis  has 
been  suggested  for  this  type  of  model,  but  it  is  unrealistic  to  expect 
our  cost  criterion  to  be  convex,  and  a piecewise  convex  analysis  does 
not  seem  to  work  for  discrete  time  models.  Therefore,  the  analysis 
must  be  performed  without  assuming  a nice  structure  on  the  cost  criterion. 
A bounding  technique  used  in  [5]  then  seems  to  be  the  only  candidate 
for  the  analysis  of  this  type  of  model. 

Lemma  3.9.  Assume  the  following  conditions  hold: 

1.  A(i)  is  nondecreasing  in  i for  0 < i < I. 

2 C(i)  is  nondecreasing  in  i for  0 < i < I. 

3.  K(s,k)  is  nondecreasing  in  s (0  £ s £ S+l)  for  k =0,1. 

U.  P > min(A(0),  C(0)). 

5.  pt(*)  c pi+1(‘)  for  0 < i < 1-1. 

Then,  for  1 < s £ S+l,  0 < i < I,  k = 0,1,  and  n > 1, 

V^(i, Sj n)  - Vk(i,s-l;n)  < M , 
a a ' - n ’ 

where 

n 

Mn  = ~~  (P  - minfA(O),  C (0 ) ) + K)  , 

and 

K = max  (K(s,k' ) - K(s-l,k’ ) ) . 
s,  k' 
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rroof . Mathematical  induction  is  applied.  For  n = 1,  it  is  easy  to 


see  that  for  0 < i < I,  l<s<  S+l,  k =0,1, 


V*(i,s;l)  - V*(i,s-ljl)  < P - minfA(O),  C(0) ) + K = M 

C X 'X  X 


Suppose  the  argument  holds  for  n = m-1  > 1,  and  consider  the  case 
for  n m.  For  k - 0 and  1 <_  s < S,  we  compare  the  corresponding 
terms  of  the  right-hand  side  of  (5.3). 


Vali’s;m))l-st  ' <Va(i’s'1;m))l-st 


A(i)  +K(s,0)  +R  (i,  s ;m-l)  - (A(i)  +K(s-1,0)  +R  (i,s-l;m-l)) 
Ct  OL 

K(s,0)  -K(s-1,0)  +0£  E p (V°(j,s;m-1)  -V°(j,s-l;m-l) 

J=0  J a 


< K(s,0)  - K ( s -1, 0 1 + a E p.  . M . < K + OM  . 
- , ij  m-1  - m-1 


J-0 


Similarly, 

= A ( i ) +K(s,0)  +E  + Gf  Q (i,s;m-l)  - (A(i)  +K(s-1,0)  + E + G + Q (i,  s-l;m-l) ) 

Q.  Q. 


(K(s,0)  -K(»-l,0)+a  E^  Pij[(l-q)(V^(j,s;m-l)-V^(j,s-l;m-l)) 

+ q(vi(j,s-l;m-l)-vi(j,s-2;m-2))] 


j=0 


■ 


for  2 < s < S 


K(s,0)  -K(s-1,0)  +a  E p ,(l-q)(vi(j,l;m-l)  - V*( j,0jm-l) ) 
, i=o  1J  a a 

for  s = 1 
I 

< R + a E p,  M . = R + aM  . 

- . - ij  m-1  m-1 

J=0 
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In  a similar  manner,  the  comparison  of  the  corresponding  third  terms  and 


that  of  the  fourth  terms  yield  the  same  upper  bound  K + oM  Hence, 

m-1 

for  l<s<S,  0 < i < I, 

V°(i,s;m)  - V°(i,s-l;m)  < K + oM  . 
ct  u — m-1 

For  s = S+l,  and  0 < i < I, 

<V°(i,S*l;„)>i_st  - <v°(i,s:m))1_st 


P + K(S+1,0)  + RQ(0,S+l;m-l) 


(A(i)  + K(S,0)  + R ( i, S; m-1) ) 


= P - A(i)  + K(s+1,0)  - K(s,0) 


+ a [ I P v°(j,s+i;m-i) 

j=0  a 


< P - A (i ) + K(s+1,0)  - K(S,0) 


+ a ^ P0j(v°fj,s+l;m-l)-v°(j,s;m-l)) 
J 


by  5 and  Lemma  3 . 1 


< P - min(A(0),  C(0))  + K + OM  , , 

m-1 


Similarly,  using  Pq(‘)  C P^C-)  and  that  V^(j,s;m-1)  is  nondecreasing 
in  i (0  < j < I),  (V°(i,S+l;m)>2_nd-  <V°( i, S; m) )£_nd  can  be  shown  to 
have  the  same  upper  bound.  Also, 

<va(l’s<l>"'»1-.t  - <vS<i>s'"»3-rd 

- P + K(S+1,0)  + Ra(0,S+l;m)  - (C(i)  + K(S,0)  + R^O,  S+l;  m-1) ) 

- min{A(0),  C(0))  + K . 
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r 


We  can  show  the  same  upper  bound  also  on  (v^(i,S+ljm) )n  V^(i, S$ m) } 
yielding  that  for  0 < i < I, 

V°(i,S+l;m)  - V°(i,S;m)  < P - mim'A(O),  C(0)}  + K + aM  . 
ji  a m-  j. 

As  P _>  minfA(O),  C^O)j  from  condition  4,  we  have  for  0 < i < I, 
and  1 < s < S+l, 

V°(i,s,m)  V°(i,s-l;m)  < P - min!A(0),  C(0))  + K + oM  = M 

a Q — m-l  m 

A similar  argument  indicates  that  for  0 < i < I,  and  1 < s < S+l, 

V1(i, s;m)  - V1(i,s-l;m)  < M , 
a ’ ’ a ’ - m ’ 

completing  the  mathematical  induction,  and  hence  the  proof.  □ 

For  the  future  use,  let 

M,  - lim  Mn  = ~ (p  ” min{A(0),  C(0))  + K). 
n -* °° 

Before  deriving  a lower  bound  or.  the  same  expression,  we  make 
the  following  remark:  If  {x^}  is  a sequence  with  the  following 

recursive  relation  starting  from  x^  = 0, 

x - min{a  + ooc  , , b]  for  n > 1, 
n n-1  — * 

then  we  have  for  0 < a < 1, 

l_an 

x - min{- a,  b)  for  n > 1 . 

n 1-a  ' - 
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-th' 


Lemma  3.9-  If  K(s,k)  is  nondecreasing  in  s (0  < s < S+l)  for 
k - 0,1,  then  for  1 < s < S+l,  0 < i < I,  k = 0,1,  and  n > 1, 


Vk(i,s,n)  - Vk(i,s-l;n)  >M  , 
a ’ a ’ n ’ 


where 


M = minfK  \ '-^-■1^'^) ) — , P - c(l)  + min  {K(S+l,k')  - K(S,k')) 
-n  - 1 - a(l-q)  . , 


and 


K - min  (K(s,k')  - K(s-l,k'))  • 
s,  k' 


Proof.  Proof  is  again  by  mathematical  induction.  For  n = 1, 

0 < i < I,  and  1 < s < S, 

V°(i,s,l)  - V°(i,s-l;l)  = K(s,0)  - K(s-1,0)  > K . 

And, 

V°(i,S+l,l)  - V°(i,S;l) 

= P - min[A(i),  C(i))  + K(S+1,0)  - K(S,0) 

> P - C(I)  + min{K(S+l,k' ) - K(S,k')}  . 
k' 

Hence,  for  k = 0,  the  hypothesis  is  true  for  n = 1.  By  a similar 
argument,  the  hypothesis  is  shown  to  hold  for  k = 1 and  n = 1. 
Suppose  the  hypothesis  is  true  for  n - m-1  > 1,  and  consider  the  case 
for  n = m.  We  compare  the  corresponding  terms  as  in  the  previous 
lemma.  Fix  i (0  < i < I).  For  1 < s < S, 
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i 


= K(s,0)  - K( s-1,0)  + a E Pij[V°(j,s;m-l)  - V°( j , s-1; m-1) ] 


J=o 


aM 


> K(S,0)  - K(s-1,0)  + a E P^.  Mn_1  > K + -^m_1  • 

d=0 


For  2 < s < S, 

(V°(i,s;«))2.nd  - <v“(i,»-i;»))2.nd 

I x 

- K(s,0)  - K(s-1,0)  + a E Pi.[(l-q)(Va(j,s;m-l) 

j=0 

+ q(V^(j,s-l;m-l) 

> K + a E p^Ul-qJM^  + qM^)  = K + OM^  • 

J *0 

For  s = 1, 

<Va(1,1;m»2-nd  - <Va(1’0i“»2-nd 

1 1 11 
r--  K(1,0)  - K(0,0)  + a E p.  ,[  (l-q)V^(j,l;m-l)  + qV  (j,Ojm-l)  - Vq(  j,Ojm-l)  ] 

J=0  J 

• K + a(l-q)M  , • 

— — -m-i 

Comparison  of  the  corresponding  third  terms  and  that  of  the  fourth 
terms  can  be  performed  in  a similar  manner,  yielding  that  for  1 < s < S, 

V°(i,s;m)  - V°(i,s-l;m)  > K + a(l-q)M(n_1  . 

For  s = S+l, 


-V^(d,s-l;m-l)) 

- V^(j,s-2jm-l))] 
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V°(i,S+l,m)  - V°(i,S;m) 

• P + K(S+1,0)  + rain{a  Z pni  VU(j,S+l;m-l) , FI  + G 

j=0 


Oj  a' 

I 3+1 


+ a Z p Z q vJ(jfs*;m-i)) 

j=0  J S’=0  * L’S  “ 


- [C(i)  + K(S,0)  + min{ct  Z pn.  V (j,S+l>m-l),  E + G 

I S+l  , 

"a/ipoj  ,4  vi,.'  vj’8',"-i))i 

= P - C(i)  + K(S+1,0)  - K(S,0) 

> P - C(I)  + K(SU,0)  - K(S,0)  . 

Therefore,  for  1 < s < S+l,  and  0 < 1 < I, 


V°(i,s;m)  - v£(i,s-ljm)  > min{K  + afl-q)^^,  P - C(l)  + K(S+1,0)  - K(S,0) } 


A similar  argument  yields  for  1 <_  s < S+l  and  0 < i < I, 
vj(i,s,m)  - V*(i,s-l;m)  >minf K + ad-q)^^,  P - C(l)  + K(S+1,1)  -K(S,1)} 


Therefore,  for  1 < s < S+l,  0 < i < I,  and  k - o, 1, 


V*(i,s;m)  - V^(i,s-l;m) 


> minfK  + a(l-q)M  .,  P - C(I)  + minf K(S+1, k' ) -K(S,k’))) 
- - -m-1 


= min(K  1 ' P - C(l)  + min{K(S+l,k'  ) - K(S,k’ 

**  1 — Of v 1 ) i.  * 


))) 


«=  M , 

-m 

which  completes  the  mathematical  induction. 


□ 
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Analogous  to  M , we  define 


= lim  Mn  = minf---  ^ y , P - C(l)  + min(K(S+l, k’  ) - K(S,k' )}) 

n ~t  OO  V 4 


It  is  easy  to  see  that 

n -n  - n+1  — n+1  — * -» 

Lemma  l . 10 . For  0 < i < I,  0 < s < S+l,  and  n > 1, 

V°(i,s;n)  - V^(i,s;n)  = K(s,0)  - K(s,l)  + 6n(i,s) 


where  -F  < &n(i,s)  < E. 


Proof.  For  0 < s < S+l,  0 < i < I,  and  n > 1, 

<V°(i,s;n))J_th  - (vj(i,s;n))._th  - K(s,0)  - K(s,l)  - F , 

for  j = 1 and  3 (j  = 1 only  if  s - S+l)  , 

<v£(i,s;n)>J-th  - (V^(i, s ;n) ) j_th  = K(s,0)  - K(s,l)  + E , 

for  j = 2 and  4 (j  = 2 only  if  s = S+l). 

Therefore, 

K(s,0)-K(s,l)-F<\£(  i,s;n)  -V^(i,s;n)  < K(s,0)  - K(s,l)  + E , 
which  implies 

-F  < 6n(l,s)  s V°(i,s;n)  - vj(i,g;n)  - (K(s,0)  - K(s,l))  < E . 
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□ 


In  this  section  we  have  failed  to  derive  sufficient  conditions 
for  the  optimality  of  a control  limit  policy  with  respect  to  a repair 
shop.  However,  it  is  possible  to  obtain  sufficient  conditions  under 
which  a control  limit  type  of  property  holds  between  two  actions 
and  a , and  between  aRC  and  a^.  The  next  lemma  is  useful  for 
this  purpose. 


Lemma  3. 11.  Assume  all  the  conditions  of  Lemma  5.8  hold,  and  furthermore, 
assume  the  following  condition  holds: 

6.  K(  s+1,0)  -K(s,0)  > K(  s+1,  1)  - K(s,  1)  +E  + F + q(Mo-Mj,  0 < s < S. 

Then,  for  0 < s < S+1,  0 < i < I,  and  n > 1,  Ra(i,s;n)  - Q^(i,s;n) 
is  nondecreasing  in  s. 


Proof.  For  1 < s < S+1,  0 < i < I,  and  n > 1, 


R^(i,s;n) 

I 

= a £ 
3=0 


- a £ 
3=0 


I 

= a £ 
J=o 


- Qa(i>s;n) 

p [K(s,0)  - K(s,l)  + 5n(j,s)  + V^(d, s;n) 

- (l-q)V^(j,s;n)  - qV^(j,s-l;n) ],  by  Lemma  3.10 

p^fKCsjO)  - K(s,  1)  + &n(j,s)  + q[  V^(  j,  s ;n)  - V^(  j,  s-1  ;n)  ] ] . 


Hence,  for  l<s<S,  0 < i < I,  and  n > 1, 
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(Ra(i> s+lj  n)  - Qa(i,s+l»n))  - (R^(i,s;n)  - 0,^(1,  s;n)) 

= a E p (K(s+1,0)  - K(s+1, 1)  +6n(j,s+l)  + q(vi(j,«+ljn)  -vJ(j,8}n)) 

j-Q  ^ u: 

- (K(s,0)  -K(s,l)  +&n(j,s)  +q(V^(j,s;n)  - V‘(  j,  s-1;  n1)  ] ] 

I 

>Ot  E p.  . 1 K( s+1,0)  - K(s+1, 1)  - (K(  s,0)  - K(  s,  1) ) - E - F + q(M  -M  )]  , 
j z=q  J n n 

by  Lemmas  3. ft,  3.9,  and  3.10 

>0,  by  6 and  since  M - M > M - M . 

— — n n — -oo  * 

For  s = 0,  0 < i < I,  and  n > 1, 

Ra(i,0;n)  - Qa(i,0jn)  = a E p [V°(j,0;n)  -V*(j,0,n)]  . 


Hence, 


(Ra(i,lin)  - Qa(i,l,-n))  - (Ra(i,0,n)  - Q^(i,0;n)) 

= a E Plj[K(l,0)  -K(l,l)  +Bn(3,l)  +q(vj(d,l,n)  -vj(j,0;n)) 
i “0 


- V°(J,0,n)  + vi(j,0;n)] 


= a E p.  [K(1,0)  - K(  1, 1)  - (K(0,0)  -K(0,1))  +6  (j,l) 
j=0  ij  n 

- 6n(J,0)  + q(vj(d,l;n)  - vJ(jf0,n))] 

I 

> a E p [K(1,0)  - K(1,1)-(K(0,0)  - K(0,1))  -F  - E + qM  ] , 


by  Lemmas  3.9  and  3.10 


> 0,  by  condition  6 and  since  Mn  > 0, 


which  completes  the  proof. 
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Notice  that  if  the  conditions  of  Lemma  3.11  are  all  satisfied, 


then  for  0 < i < I,  E (i,s)  - QMi,s  is  also  nondecreasing  in  s for 
0 < s < S+l. 

Suppose  that  all  the  conditions  in  both  Lemmas  3.2  and  3.11  are 
satisfied.  By  Lemma  3.2,  there  is  a stationary  control  limit  policy 
with  respect  to  an  operating  machine  which  minimizes  the  total  expected 
r-discounted  cost.  Consider  the  structure  of  the  optimal  policy  further, 
and  we  find  that  its  i-s  diagram,  as  is  seen  in  Fig.  3.3  is  divided  into 
upper  and  lower  divisions  for  each  k.  The  action  of  leaving  an  operating 


iX  0 1 2 ...  S S+l 


“lo  or  V 


machine  in  operation  is  taken 
in  each  state  in  the  upper 
region,  and  the  action  of 
repairing  an  operating  machine 


^RO  °r  aRC 


Figure  3.3.  A Typical  Optimal  Control 
Limit  Policy  with  Respect 
to  an  Operating  Machine. 


is  taken  in  each  state  in  the 


lower  region.  The  former  has 


two  alternatives 


a^r,  while  the  latter  has  two 


alternatives  aRQ  and  aRC> 


We  now  focus  on  the  possibility  of  subdividing  each  region  having  two 
alternatives.  When  the  state  of  the  system  at  the  beginning  is  (i,s,k) 
with  k - 0,  0 < i < I,  and  0 < s < S,  the  difference  between  the 
total  cost  of  choosing  a^  at  the  beginning  followed  by  the  best 
policy  and  that  of  choosing  a^  at  the  beginning  followed  by  the  best 
policy  is 


A(i)  + K(s,0)  + Ra(i,s)  - (A(i)  + K(s,0)  + E + G + Q^i.s)) 
= R^i.s)  - Qa(i,s)  - (E  + G)  , 


6l 


which  is  nondecreasing  in  s (0  < s < S)  by  Lemma  3.11.  For  k = 1, 

the  corresponding  difference  becomes  R^(i,  s)  - Q^(i,s)  + (F-G), 

which  is  also  nondecreasing  in  s (0  < s < S) . Therefore,  there  exist 

critical  numbers  sL  for  each  fixed  k (k  = 0,1)  and  i (0  < i < I) 

1 , k 

such  that  for  all  (i,s,k)  with  s < a^  is  better  than  a^, 

and  for  all  (i,s,k)  with  s > s^  a^0  is  no  worse  than  a^. 

This  implies  that  the  upper  division  where  a or  a^  is  optimal 

can  be  divided  into  left  and  right  subdivisions,  a^  is  optimal  in 

each  state  in  the  left  subdivision,  while  a^  is  optimal  in  each  state 

in  the  right  subdivision.  In  a similar  manner,  we  can  show  that  there 

exist  critical  numbers  sV  , for  each  fixed  k and  i such  that  for 

i,k 

all  (i,s,k)  with  s < s.  ^ aRC  is  better  than  a^,  and  for  all 

(i, s,k)  with  s > s^ 

— l,  K 

can  be  divided  into  two  subdivisions.  There,  a is  optimal  in  each 

KG 

state  in  the  left  subdivision,  while  a^,.  is  optimal  in  each  state  in 
the  right  subdivision.  We  call  this  type  of  policy  a stationary  two- 
dimensional  weak  control  limit  policy.  One  realization  of  a two- 
dimensional  weak  control  limit  policy,  optimizing  our  problem,  is 
shown  in  Fig.  5.U. 

The  control  limits  found  in  this  kind  of  policy  are  those 
on  the  action  of  repairing  or  leaving  an  operating  machine,  those  on 
the  action  of  a^  or 

a . Control  limits  on  the  action  of  opening  or  closing  the  repair 
RO 

shop  might  not  exist.  In  this  sense,  this  type  of  policy  is  weaker 


a^,  and  those  on  the  action  of  aR(,  or 


, aRQ  is  no  worse  than  a^,.  The  lower  division 


than  a two-dimensional  control  limit  policy. 


k - 0 (closed)  k 1 open) 


Figure  3. A Typical  Optimal  Two-Dimensional  WeaK  Control  Limit  Policy. 

As  a conclusion  of  this  section,  we  restate  the  above  discussion 
as  a theorem 

Theorem  3, 1C.  Assume  the  following  conditions  hold; 

1.  C(i)  is  nondecreasing  in  i for  0 < i < I 

2.  A( i ) - C(i  i;  nondecreasing  in  i for  0 < i < I. 

3.  K's.k  is  nondecreasing  in  s (0  < s < S+l)  for  each  k = 0,1. 
k.  P > minf A(0  , C(0) }. 

5.  P^  ) Pi+1(  ) for  0 < i < 1-1. 

6.  Ki.s+l.C  - K(s,0)  > K s+l,  1) -K;  s,  1)  + E + F + q(Mw  - , where 

M,  « ~ 1P  ' min  A(0),  C(0)J  + K] 

K 

^ min-r~5a-'q)  > P - C(l)  + min  (K(S+1,  k')  - K(S,k'))) 

K max  (K(s,k')  - K(s~l,k’)j 

s,k' 

K = min  {K(s,k')  - K(s-l,k’)}  . 
s,  k' 

Then  there  exists  a stationary  two-dimensional  weak  control  limit  policy 
which  minimizes  the  total  expected  a-discounted  cost  of  the  model. 
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5.5.  Exchange  Options 


In  the  previous  sections,  the  role  of  the  spare  machines  was 
to  supply  a machine  for  operation  when  the  previously  operating  machine 
was  chosen  to  be  repaired.  Therefore,  as  long  as  the  repair  decision 
was  not  selected,  we  did  nothing  with  the  spare  units  even  if  they  were 
in  better  condition  than  the  operating  machine.  In  this  section,  the 
action  space  is  expanded  to  allow  for  the  case  of  exchanging  an  operating 
machine  with  one  of  the  available  spare  units  at  any  time  period  so  that 
the  operating  machine  can  always  be  in  better  condition  than  any  spare 
unit.  The  option  available  at  any  period  is  to  exchange  an  operating 
machine  with  one  of  the  available  spare  units,  or  to  send  an  operating 
machine  to  the  repair  system  and  introduce  one  of  the  available  spare 
units  as  a new  operating  machine.  A decision  to  keep  an  operating 
machine  in  operation  is  considered  as  a decision  to  exchange  an  operating 
machine  with  itself.  Machines  sent  to  the  repair  system  are  repaired 
in  one  period  when  the  repair  service  gate  is  open,  and  repaired 
machines  are  available  as  spare  units  in  their  best  conditions.  The 
cost  structures  are  the  same  as  before.  An  exchange  action  may  take 
place  at  no  extra  cost.  The  structure  of  an  optimal  policy  minimizing 
the  total  expected  a-discounted  cost  is  of  interest. 

In  order  to  describe  the  model,  we  must  specify  the  number  of 
spare  machines  in  the  i-th  condition  for  each  0 < i < I,  since 
spare  units  are  no  longer  guaranteed  to  be  in  the  best  condition.  Let 
w = ( i, s, Sq, . . . , s j, k)  denote  the  state  of  the  system  where  there  is  an 
operating  machine  in  the  i-th  condition,  the  number  of  machines  in  the 


6b 


r 


t 


r 


repair  system  is  s,  the  number  of  spare  units  in  the  j-th  condition 

is  s.  (0  < j < I),  and  the  state  of  the  repair  service  gate  is  k. 
J 

Necessarily, 

I 

0 < i < ;[,  s > 0,  s . > 0 (0  < j < D,  s + Z s = S. 

3 3=0  3 


We  denote  (-,S+1,0,0, . . . ,0,0)  to  represent  the  state  where  no 
operating  machine  is  available. 

For  w = (i,s,s(),...,sI,k),  let 

U(w)  = U|s.  > 1,  0 < j < I)  . 

There  is  at  least  one  spare  unit  in  the  j-th  condition  if  j € U(w). 
Define  the  following  actions: 

a^,  . : exchange  an  operating  machine  for  a spare  unit  in  the  j-th 

b,  j 

condition. 

aD  . : send  an  operating  machine  to  the  repair  system  and  introduce 

J 

a spare  unit  in  the  j-th  condition  as  a new  operating  machine, 
a^  : send  an  operating  machine  to  the  repair  system. 

aC  ^a0^  : close  (open,  respectively)  the  repair  service  gate. 

The  action  space  A(w)  of  this  system  depends  on  the  state  of 
the  system  w = (i, s, sQ, . . . , Sj, k) , and  can  be  represented  as 

A(w)  = A^w)  X A2(w)  , 


where  for  U(w)  j 0,  A1(w)  = {a^,  aR^  | j 6 U(w)  U { i) , j'  € U(w)}, 
and  for  U(w)  = 0 and  s = S,  A^(w)  = { a^  ),  and  A'(w)  = 0 
if  s = S+l,  and  A2(w)  = (a^a^). 
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Let  Va(i, 8, 3^, . . . , Sj ;n)  be  the  minimum  n period  expected 
a-discounted  cost  starting  from  state  (i,s, sQ, . . . ,s  ,k) . Then  we  have 
the  following  recursive  equation: 

For  U(w)  / 0,  0 < i < I,  0<s<S,  and  n > 0, 

Va(i,s,s0' 


= min{  min  f A ( i ) + K(s,0)  + a Z p.., 


jtU(w)  U{  i} 


J'=0 


jj' 


0, 


min  { C(i)  + K(s,0)  + a Z p 


j£U(w) 


j'=o 


Jo' 


•V°(d ' > s+1,  Sq,  • . . , Sj-1, . . . , S.J.J  n) ) , 


min  {A(i)  + K(s,0)  + E + G + a Z p.,, 

j£U(w)U{i}  j'=o  JO 

^^0  so+s>  • • • » s j“l>  • • • > s^+1, . . . , Sjj  n)  ] , 

I 

min  {C(i)  + K(s,0)  + E + G + a Zp... 


J€U(w) 


j'=0 


jj' 


’ ' »°>  so+s+1»  si>  • * ’ ’ s j ~l>  ■ • • > si>  • • • » Sji  n) } ] . 


Va(  » s,  Sq,  ...,s.,...,Sjj  n+1 ) 


min{  min  { A( 1)  + K(s,l)  + F + a Z p,,, 


JGU(w)U{i) 


J'=0 


JJ' 


> s » Sq,  . . . , s -1, . . . , s^+1, . , . , s jj  n,1 } , 
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I 

min  {C(  i)  + K(s,  1)  + F + a T.  p . . , 

J€U(w)  J'=0 

•V°(j',s+l,s0,...,s.-l,  ...,sI;n)}  , 

I 

min  (A(i)  + K(s,l)  + G + a T.  p.., 

jtU(w)  U{ i)  J'=0 


(J* 


,0,s0+s,s1. 


s .-1, . . . , s . +1, . . 
3 i 


I 

min  (C(i)  + K(s,l)  + G + a E p.., 
jfcU(w)  j’=0 


■V^(j,,°,s0+s+l,s1,  • . . ,s  , 


>SI 


For  U(w)  = s = 3,  0 < i < I,  and  n > 0, 
V°(i,S,0,...,0;n+l) 


= min(A(i)  + K(S,0) 
C(i)  + K(S,0) 
A(i)  + K(S,0) 
C(i)  + K(S,0) 


+ a Z P.,,  v°(j',s,0,...,0jn) 

j'=0  J 

+ a V°(-,S+l,0,...,0;n), 

I i 

+ e + g + a ZP  v (j*,o,s,o 

j'=0  J 

+ E + G + aV*(0,0,S,0, ...,0;n)) 
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J 


• y Sjj  n) ) , 


; n) } } . 


, . . . ,0;  n) , 


I 


I 


i 


V*(i,S,0,...,0;n+l) 

1 0 

= mint A(i)  + K(S,1)  + F + a ..  P.  V (j ’ ,S,0, . . . ,0; n) , 

j ' =0  1J 

C(i)  + K(S, 1)  + F + aV°(-,S+l,0,...,0;n)  , 

I 

A ( i)  + K(S, 1)  + G + a Z p...  V (j\0,S,0>...,0;n)  , 

j ' =0  J 

C(i)  + K(S, 1)  + G + aVL(0,0,S,0, . . .,0jn) } . 

0t 

For  U(w)  = 0,  s = S+l,  and  n > 0, 

V°(-,S+1,0,  ...,0;n+l)  --  min{  P + K(S+1,0)  + aV°  (-,  S+l,  0, . . . ,0;  n) , 

P + K(S+1,0)  + E + G + aV^(0,0, S,0, . . . ,0;n) } . 

V^(-,S+1,0, . ..,0}n+l)  = min{  P+K(S+1,1)  +F  + aV°(-,S+l,0, . . .,0;n), 

P + K(S+1,1)  + G + aV^(0,0,S,0,...,0>n))  . (5. it) 

This  model  is  much  more  complicated  than  the  previous  models 
without  the  exchange  options.  Fortunately,  the  analysis  on  the  structure 
of  an  optimal  policy  can  be  performed  in  a similar  manner. 

Lemma  $ . 1' . Assume  the  following  conditions  hold: 


1. 

c(i) 

is  nondecreasing  in 

i for  0 < i < I. 

2. 

A(i) 

is  nondecreasing  in 

i for  0 < i < I. 

5. 

Pi(-) 

CP1+1(0  for  0 < 

i < 1-1. 

Then  for  n > 1,  and  k =0,1, 
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k 

(a)  Va(i,s,sQ, . ..,s  .+1, . . .jSjjn)  is  nondecreasing  in  j (0  < j < I) 
for  0 < i < I,  and  for  each  fixed  (s, sQ, . . . , s^)  with  s > 0, 


s.  > 0 (0  < h < I)  and  s + Z^^  sh  = S-l. 


(b)  V*(i,  s,  Sq,  . . . , Sj.;  n)  is  nondecreasing  in  i (0  < i < I)  for  each 
fixed  (s,s0,...,s)  with  s > 0,  sh  > 0 (0  < h < I)  and 

• + SU  sh ' s- 


Proof.  Mathematical  induction  is  applied  simultaneously  on  both  (a) 
and  (b).  Both  (a)  and  (b)  hold  for  n = 1.  Suppose  the  claims  (a) 
and  (b)  both  hold  for  n = m > 1.  Then  from  (3-M,  for 
w = ( i , s , Sq  ,••••> Sj)...,Sj,0)j 


( VQ(i,  s>  Sq»  ...» Sj+1, , s j.; m+1 ) ) ^_g^. 


= A(i)  + K(s,0)  + min{a  Z p^  , VQ( j’ , s, sQ, . . . , s.., . . . , Sj+1, . . . , s^i 

J 


Q 

min  (a  Z p.  . , V ( j 1 , s,  s_, . . . , s, -1, . . . , s +1, 


h£U(w)U{i)  j'=0 


hj ' 'a 


. . . , s^+1, . . . , s^; m) ) } . 


Now  by  the  inductive  hypothesis  on  (b)  for  n = m,  and  by  condition  3, 
Zj1=0  Pjj,  V^(d',s,s0,...,sJ,...,s1+l,...,s][;m)  is  nondecreasing  in 
j (0  < j < I).  Also  by  the  inductive  hypothesis  on  (a)  for  n = m, 
V°(j',s,s0, ...,«h-l, ...,Sj+l, ...jSj+l, ...jSjjm)  is  nondecreasing  in 
j (0  < j < I).  Therefore,  the  first  term  of  V^(i, s, sQ, . . . , s^.+l, 

. ..,Sjjm+l)  is  nondecreasing  in  j (0  < j < I).  In  a similar  manner, 
other  terms  are  shown  to  be  nondecreasing  in  j (0  < j < I),  yielding 


that  V^( i, s, Sq, . . . , s >1, . . . , Sjjra+l)  is  nondecreasing  in  j (0  < j < I), 
The  sane  argument  follows  for  V^(  i,  s,  sQ , . . . , s^  +1, . . . , s^jm+l) , so  that 
(a)  holds  for  n = m+1. 

For  U(w)  t with  w = (i,  s,  sQ, . . . , Sj.,0) , 


(Vi,s>  V’*',SI'm+1))l-st 


= A(i)  + K(s,0)  + min{  min  a £ p.,,  V°(  j ' , s,  s , . . . , s -1, . , 

jeu(w)  j’=0  a o J 

• • • 9 s ji  in) ) 


• y 


Q!  Tj  > s>  sq>  •••  >s ^9  •••  > s^;m) ) . 

j *0 


By  the  inductive  assumption  on  (a)  for  n = m,  V^( j, s, s^, . . . , Sj-1, 

. . . • . . , s^jm)  is  nondecreasing  in  i,  and  by  the  inductive 
assumption  on  (b)  for  n = m and  by  condition  3,  E*  p V°(j',s, 
sQ, ...,s  ;m)  is  nondecreasing  in  i (0  < i < I).  Hence,  the  first 
term,  of  V^(i, s,  sQ, . . . , s^m+l)  is  nondecreasing  in  i (0  < i < I), 
using  condition  2.  The  same  argument  follows  for  the  third  term. 

For  the  second  and  the  fourth  terms,  notice  that  they  depend  on  i 
only  through  C(i),  and  hence,  they  are  nondecreasing  in  i (0  < i < I) 
by  condition  1.  Therefore,  V^(i, s, sQ, . . . . , Sjjtn+1)  is  nondecreasing 
in  1 (0  < i < I)  for  U(w)  / (j/.  In  a similar  manner, 

V^(i, s, sQ, . . . , Sjjm+1)  can  be  shown  to  be  nondecreasing  in  i 

(0  < i < I)  for  U(w)  / (jl.  For  U(w)  = <jl,  the  only  possibility  to 

be  considered  is  the  case  (s, sQ, . . . , s^)  = (S,0,...,0),  and  it  is 
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clear  that  V^(i,S,0, . . . ,0;m+l)  is  nondecreasing  in  i (0  < i < I)  for 
k = 0,1  from  (3.4).  Therefore,  (b)  also  holds  for  n = nat-1,  completing 
the  mathematical  induction  and  the  proof.  □ 

Lemma  3. 14.  Assume  the  following  conditions  hold: 

1.  K(s,k)  is  nondecreasing  in  s (0  < s < S+l)  for  k =0,1. 

2.  P > C(I). 

Then  for  k = 0,1,  0 < i < I,  l<m<S,  0 < s < m-1,  Z*  , s.  = S-m, 

J ’ * J 

s.  > 0 (0  < j < I),  and  n > 1, 

J 

It  It 

Va  ( i , s , m— s,  s^,.*.,Sjjn)  < (i,  s+l, m-s-1,  s^, . • . , s j j n . 

Proof.  Mathematical  induction  is  again  applied.  From  (3.4),  it  is 
clear  that  the  claim  holds  for  n = 1.  Suppose  it  holds  for  n (>  1), 
and  consider  the  case  for  n+1.  Comparisons  of  the  corresponding  terms 
of  the  right-hand  side  of  (3-4)  will  give  the  desired  result.  For 
example,  we  compare  the  corresponding  second  terms.  Let 
w = (i,  s,m-s,  s^, . . . , 3^,0) , and  let  w'  = (i,  s+l, m-s-1,  s , . . . , s.j.,0) . 

Then  for  l<m<S,  0 < s < m-2,  and  for  1 < m < S-l,  s = m-1, 

(V°(i,s,m-s,s1, . •.,sIJn+l))2_nd 

1 0 

= min  (C(i)  +K(s,0)  +a  I p V ( j ' , s+l,m-9,  s , . . . , s -1, . . . , s 5 n) ) 
j€U(w)  j’=0  33  a 1 3 1 

1 0 

min  (C(i)  + K(s+1,0)  + a L p V ( j ' , s+2,  m-s-1,  s , . . . , s -1, . . . , s j n) ) , 
_jeu(w)  J'=0  33  a 1 3 1 

by  condition  1,  inductive  hypothesis,  and  U(w' ) C U(w) 
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= (V°(l,s+l,m-s-l,s1, ...,sI;n+l))2_nd  . 

Notice  that  if  conditions  U and  5 are  satisfied, 

V°(-,S+l,0,..,,0jn)  - V°(i,S,0, . . . ,0, n)  > P - C(i)  + K(S+1,0)  - K(S,0)  > 0, 

for  any  0 < 1 < I.  Therefore,  for  m = S,  s = S-l,  and  0 < i < I, 

(V°(i,S-l,l,0,...,0,n+l))2_nd 

1 0 

= C ( i ) + K(S-1,0)  + a Z Pn,,  V^(j',S,0,0,...,0;n) 

j'=0  J 

< C(i)  + K(S,0)  + aV°(-,S+l,0,0, ...,0;n) 

= (V°(i,S,0,...,0;n+l))2_nd  . 

Thus,  the  claim  holds  for  the  corresponding  second  terms  for  n+1.  In 
a similar  fashion,  the  claim  can  be  proved  for  other  corresponding 
terms,  and  also  for  the  case  k = 1.  □ 

Suppose  all  the  conditions  in  Theorem  3.7  hold.  Then  we  notice 
that  both  Lemmas  3.13  and  3.1^  hold.  Furthermore,  notice  that  as 
A(i)  - C(i)  is  nondecreasing  in  i (0  < i < I),  and  since  both  the 
second  and  fourth  terms  of  the  right-hand  side  of  V^(i, s,s  , ...,Sjjn) 
in  (3.M  depend  on  i only  through  C(i), 

min{  { VQ( i,s,s^, . . . , s^; n)  / ( ^ot' ^,s,so’  ’ ' ^3-rd^ 

k> 

- min(  ( V^( i,  s , Sq,  , . . , s n) ) (V^( i,  s,  s^, . . . , n) ) 
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is  nondecreasing  in  i (0  < i < I)  by  Lemma  3.13.  This  provides  the 

following:  For  each  fixed  (s,  sQ, . . . , s^.,  k) , there  exists  a critical 

* 

number  on  i,  say  i depending  on  (s, sQ, . . . . , s^ , k) , such  that  for 

# 

all  states  with  i < i , exchanging  an  operating  machine  for  a 

* 

suitable  spare  unit  is  optimal,  and  for  all  states  with  i > i , 
repairing  an  operating  machine  (or  sending  it  to  the  repair  shop) 
and  introducing  a suitable  spare  unit  is  optimal.  Similarly,  if 
Lemma  3.1^  holds,  then 

min((V^li,s,m-s,s1, .. .,sI;n)>1_st,  <V^(i, s,m-s, s^ . . . , Sj ;n) >2_nd) 
k k 

- min((Va(i,s,m-s,s1, . . . , Sj ;n) >5_rd>  <v£(i, s, m-s, s^  . . . , Sj ;n) ) ^_th} 

is  nondecreasing  in  s (0  < s < m),  since  both  the  third  and  fourth 

terms  depend  on  s only  through  K(s,k).  This  provides  the  following: 

For  each  fixed  m (l  < m < S),  0 < i < I,  and  (s^, . . . , s ) with 

Zj  ^ s . = S-m,  s^  > 0 (l  < J < I),  there  exists  a critical  number  on  s, 
* 

say  s depending  on  all  m,  i,  and  (s^, ...,Sj),  such  that  for  all 

li- 
the states  with  s < s , closing  the  repair  service  gate  is  optimal, 

and  for  all  the  states  with  s > s , opening  the  repair  service  gate 

is  optimal.  This  enables  us  to  divide  each  i-s  diagram  with  fixed  m, 

k and  (s^, ...,s_)  into  U regions  as  shown  in  Fig.  3.5*  Moreover,  the 

optimal  region  for  the  decision  of  repairing  an  operating  machine 

and  opening  the  repair  service  gat-  xs  shown  to  be  rectangle  as  in 

the  corresponding  model  without  exchange  options. 
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Thus  far,  the  structure 
of  an  optimal  policy  for  this 
problem  resembles  that  of  the 
simplified  maintenance  model 
with  control  of  queue,  where 
a two-dimensional  control 
limit  policy  is  assured  to 
be  optimal.  But  we  must 
realize  that  the  statement  "exchanging  is  optimal"  implies  only  that 
exchanging  an  operating  machine  for  a suitable  spare  unit  is  optimal, 
and  it  does  not  specify  which  spare  unit  should  be  introduced.  Similarly, 
there  remains  an  ambiguity  when  it  is  said  that  "repairing  is  optimal." 

It  seems  reasonable  to  consider  that  a spare  unit  whose  condition  is 
the  best  among  all  the  available  spare  units  should  be  introduced  to 
minimize  the  total  cost,  but  it  has  yet  to  be  proved.  What  we  can  see 
from  the  structure  of  the  formulation  of  the  model  are  the  following: 

In  the  repair-close  optimal  region,  the  condition  of  a spare  machine 
to  be  introduced  as  an  operating  machine  does  not  depend  on  the  con- 
dition of  the  previously  operating  machine.  In  the  exchange-open 
optimal  region,  the  condition  of  a spare  machine  to  be  exchanged  with 
an  operating  machine,  does  not  depend  on  the  number  of  machines  in 
the  repair  system.  In  the  repair-open  optimal  region,  the  condition 
of  a spare  machine  to  be  introduced  depends  neither  on  the  number  of 
machines  in  the  repair  system  nor  on  the  operating  condition  of  the 
previously  operating  machine. 


Figure  3.5.  A Typical  Optimal  Policy 
with  Exchange  Options. 


3.6.  Computational  Remarks 

All  the  models  treated  in  this  paper  are  Markov  decision 
models,  and  the  usual  techniques  to  compute  an  optimal  policy  for  the 
Markov  decision  process  such  as  the  policy  improvement  procedure  and 
the  LP  approach  are  applicable  here  too.  However,  when  an  optimal 
policy  is  known  to  possess  some  special  simple  structure,  it  seems 
reasonable  to  explore  whether  the  computational  work  will  be  signi- 
ficantly reduced  if  the  search  for  an  optimal  policy  can  be  performed 
among  the  policies  with  the  same  simple  structure.  A policy  improve- 
ment algorithm  among  stationary  control  limit  policies  has  been  intro- 
duced in  [ 5] , but  the  conditions  needed  for  the  application  of  the 
algorithm  are  too  restrictive  to  be  applied  to  our  models,  A policy 
improvement  hybrid  algorithm,  seen  also  in  [5],  would  be  a suitable 
method.  In  this  algorithm,  better  policies  are  searched  iteratively 
among  a set  of  policies  with  the  desired  special  structure  as  long 
as  it  is  possible,  and  then  usual  policy  improvement  procedure  is 
applied  to  obtain  an  optimal  policy.  Although  this  algorithm  has  been 
investigated  to  obtain  an  optimal  simple  structured  control  limit 
policy,  it  can  be  easily  modified  to  solve  more  complicated  models 
such  as  our  models  treated  here. 
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3.7.  Conclusions 


In  this  paper,  two  different  kinds  of  discrete  time  maintenance 
models  with  repair  shops  have  been  studied.  In  each  model,  sufficient 
conditions  have  been  obtained  under  which  the  optimality  of  some  kind 
of  control  limit  policy  is  assured. 

The  first  type  of  model  has  a multiple  number  of  repair  shops 
in  the  system.  When  information  on  the  type  of  repair  required  is 
available  at  the  moment  of  each  decision,  simple  optimal  maintenance 
policies  are  obtained  under  the  mild  conditions.  On  the  other  hand 
very  restrictive  conditions  are  required  for  the  case  when  such 
information  is  not  available. 

In  the  next  model  considered,  the  action  space  was  expanded 
to  include  the  option  of  opening  or  closing  the  repair  service  gate, 
as  well  as  the  option  of  repairing  or  leaving  an  operating  machine 
alone.  A two-dimensional  control  limit  policy  was  defined,  and  the 
optimality  of  such  a control  limit  policy  was  assured  when  the  repair 
time  of  a machine  is  negligibly  small,  a very  reasonable  condition. 

When  the  repair  time  is  not  negligible,  but  has  a geometric  distribution, 
a restrictive  condition  on  the  holding  costs  must  be  satisfied. 

Since  the  discrete  time  queueing  control  problem  has  not  been 
fully  studied,  there  are  several  extensions  that  can  be  made  on  our 
second  control  of  queue  model.  Controlling  the  queue  length  by 
changing  the  repair  service  rate,  controlling  a multiple  number  of  repair 
service  stations  by  opening  or  closing  them,  or  assuming  the  repair  time 
distribution  is  other  than  geometric  are  some  topics  for  future  research. 
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